Skip to content

Deployment Guide

This document outlines the deployment strategies for the project, covering both Cloud Run and Heroku deployment options.

Cloud Run based deployment

Prerequisites

For Cloud Run deployment, the project must include a Dockerfile in the root directory. This Dockerfile is used by Google Cloud Build to containerize the application during the CI/CD process.

Branch Strategy

The project uses a three-branch deployment strategy:

  • develop - Development environment
  • staging - Staging environment
  • main - Production environment

CI/CD Pipeline

The CI/CD process is handled by Google Cloud Build, which automatically builds and deploys the application when changes are pushed to the respective branches. This follows a GitFlow workflow where commits to the protected branches trigger automatic deployments.

CircleCI Checks for Dev Deployments

For projects with CircleCI enabled, CircleCI checks must be complete before triggering Cloud Build deployment in the dev environment. Note that not all projects have CircleCI enabled, and this requirement only applies when CircleCI is configured for the project.

Manual Deployment via Cloud Build

You can manually trigger a deployment through Cloud Build. In the triggers section, you can click on the Run button, which will trigger a build in the specified branch.

Important use case: Manual deployment via Cloud Build is particularly useful when you need to force a deployment even if the CircleCI build fails. This allows you to bypass CircleCI checks and deploy directly to the development environment when necessary.

Monitoring and Management

Deployment Trigger Configuration

Deployment triggers can be created and managed in the Google Cloud Console at: https://console.cloud.google.com/cloud-build/triggers

Build Progress Tracking

Monitor the progress of CI/CD builds at: https://console.cloud.google.com/cloud-build/builds

Changing Environment Variables

Go to Cloud Run service configuration, select "Edit & deploy new revision" and then "Variables & Secrets"

Restarting Application

Cloud Run doesn't have a built-in restart feature. However, you can achieve a restart by editing the service's YAML configuration, for example at (https://console.cloud.google.com/run/detail/us-central1/%3Capp_name%3E/yaml/edit). To trigger a restart, create or change the value of the spec.template.metadata.labels.cred-restart-count label. This will force Cloud Run to create a new revision, effectively restarting the application.

Workload Monitoring

Track application performance and resource usage at: https://console.cloud.google.com/run/detail/us-central1/%3Capp_name%3E

Note: Replace <app_name> with your actual application name

Application Logs

View and analyze application logs at: https://console.cloud.google.com/logs/query;query=resource.type%20%3D%20%22cloud_run_revision%22%0Aresource.labels.service_name%20%3D%20%22%3Capp_name%3E%22

Note: Replace <app_name> with your actual application name

Executing Adhoc Commands

The following applications have execution scripts available for running adhoc commands:

  • cred-model-api has execute-cred-model-api-cmd.sh
  • cred-api-commercial has execute-cred-api-commercial-cmd.sh
  • cred-filter-api has execute-cred-filter-api-cmd.sh

These scripts can be used like:

CRED_ENV=dev ./execute-cred-model-api-cmd.sh "node dist/data/clear-redis-cache.js"

To execute a one-shot task, these scripts will create a Kubernetes job that will run the command. The CRED_ENV variable specifies the environment, which can be dev, staging, or prod. You can monitor job execution via logUrl from script output or via the script output (kubectl logs)

Note: To use the deployment and adhoc command scripts described below, ensure you have both the gcloud and kubectl command line tools installed and configured on your machine.

Cloud Run Applications

This section lists the most relevant applications deployed in Cloud Run. Each application typically includes separate environments for development, staging, and production, with direct links provided for monitoring services and logs. We also use Cloud Run jobs for the release process, where we typically publish the GraphQL subgraph and apply database migrations.

cred-api-commercial

The cred-api-commercial service is deployed to Cloud Run with the following environments:

cred-model-api

The cred-model-api service is deployed to Cloud Run with the following environments:

cred-filter-api

The cred-filter-api service is deployed to Cloud Run with the following environments:

cred-agent-ai

The cred-agent-ai service is deployed to Cloud Run with the following environments:

Rate Limiting

We use rate limiters in commercial-api and model-api to protect those services from excessive traffic. You can allow specific accounts to bypass these limits by configuring the API_RATE_WHITELIST_EMAILS environment variable with a comma-separated list of allowed email addresses. To see or change the current value per environment, use the Cloud Run environment config (Variables & Secrets are in the YAML):

The web app is fronted by the Vercel firewall, which can also be configured (e.g. to allowlist IPs or adjust rules) and may affect traffic before it reaches our APIs.

If you see errors in the GraphQL layer such as HTTP 429 (Too Many Requests) or messages like HTTP fetch failed from "commercial-api": 429: Too Many Requests (or similar for model-api), they are coming from the rate limiters configured in those subgraphs. In that case, consider adding the affected account to API_RATE_WHITELIST_EMAILS for the relevant environment, or review the Vercel firewall and subgraph rate-limit configuration.

Vercel Firewall

The cred-web-commercial project is the only one with a Vercel firewall enabled. It covers all traffic across production, staging, develop, and preview environments.

Protected Domains

Domain Environment
commercial.credplatform.com Production
commerical-prod-vercel.credplatform.com Production (alias)
cred-web-commercial-credai.vercel.app Production (auto-assigned)
cred-web-commercial-git-main-credai.vercel.app Production (auto-assigned)
commercial-staging.credplatform.com Staging
commercial-staging-vercel.credplatform.com Staging
commercial-dev.credplatform.com Develop
commercial-dev-vercel.credplatform.com Develop
cred-web-commercial-env-develop-credai.vercel.app Develop
cred-web-commercial-git-*-credai.vercel.app Preview (per-branch)

How Requests Are Evaluated

When a request arrives, Vercel evaluates it through three layers in order:

  1. Custom Rules — evaluated first; can bypass the entire firewall
  2. Managed Rules — Vercel-maintained bot detection
  3. Core Rule Set (CRS) — OWASP-style attack pattern detection

Layer 1: Custom Rules — QA Bypass

A request is bypassed if it matches either of these two condition groups in the current API configuration:

  1. The request includes the header:
  2. x-vercel-protection-bypass:
  3. To find the token, search "vercel bypass token" in 1Password.
  4. A single Host group containing both conditions:
  5. Host header equals commercial-dev.credplatform.com
  6. Host header equals commercial-staging.credplatform.com

Warning: In the current API rule structure, the dev and staging Host conditions are in the same condition group. If Vercel evaluates conditions in a group with AND logic, that Host group cannot match a single request (a request cannot have two Host values simultaneously). As currently configured, bypass is therefore effectively header-based (x-vercel-protection-bypass) until the Host matching structure is adjusted.

Layer 2: Managed Rules

Rule Action Description
Bot Protection Challenge Analyzes requests using TLS fingerprinting (JA4), behavioral heuristics, IP reputation, and known bot signatures. Suspected bots are presented with a challenge; those that fail are blocked. This is the only rule actively blocking traffic.
AI Bots Log Identifies known AI crawlers (GPTBot, ChatGPT-User, CCBot, Google-Extended, etc.). Requests are logged but not blocked. Change action to "deny" or "challenge" to start blocking.

Layer 3: Core Rule Set (CRS)

All four CRS categories are in log-only mode — they detect and record matches but do not block requests.

Category What It Detects
General (gen) HTTP protocol violations, malformed requests, request smuggling, generic attack signatures
XSS Injected JavaScript/HTML — <script> tags, event handlers, javascript: URIs, encoded variants
SQLi SQL fragments — UNION SELECT, OR 1=1, comment sequences, stacked queries
RCE OS command injection — shell metacharacters, /bin/sh, cmd.exe, eval

Disabled Rules — Enable If Malicious Activity Is Detected

Two additional custom rules exist but are currently disabled. They should be enabled manually if malicious activity is detected:

Rule Action Description
Rate Limit Rate limit Throttles excessive requests from a single source. Enable this to mitigate abuse, scraping, or denial-of-service attempts.
signing challenge Challenge Current match path is /_next/* (Next.js static assets), not sign-in routes. Verify and update this rule if sign-in protection is intended.

To enable either rule, go to the Vercel Firewall settings for cred-web-commercial and toggle the rule on. No redeployment is required; changes take effect immediately.

Summary

Layer Rule Blocks Traffic?
Custom Rules Bypass QA Testing No — allows through
Managed Rules Bot Protection Yes — bots that fail the challenge are blocked
Managed Rules AI Bots No — log only
CRS General, XSS, SQLi, RCE No — log only

In practice: Bot Protection is the only rule actively blocking traffic. The CRS and AI bot rules are in observation mode. In the current custom-rule structure, bypass is effectively header-based, and dev/staging custom domains are not automatically bypassed by Host matching alone.

Troubleshooting

  • If legitimate users or automated tools are being challenged or blocked, verify whether the request includes the x-vercel-protection-bypass header. With the current custom-rule structure, this header path is the effective bypass mechanism.
  • If you see HTTP 429 errors, those are not from the Vercel firewall — they come from the rate limiters in commercial-api or model-api (see the section above). The Vercel firewall does not currently enforce rate limits.
  • To change CRS or AI bot rules from log to block, update the action in the Vercel firewall dashboard for the cred-web-commercial project.

Task Scheduling

We support two complementary scheduling/execution modes depending on the workload profile:

1) Cloud Scheduler → Kubernetes Job (long-running/batch)

2) Cloud Run workers with Redis queue (short-running/async)

  • Use when: The task is short-running and is typically enqueued by the web application.
  • How it runs: The web app enqueues tasks into Redis; worker processes (deployed on Cloud Run) consume and execute them.
  • Console: Worker Pools · Project-specific worker pools list: cred-1556636033881

    Note: The application must implement a queueing subsystem (such as BullMQ) to facilitate the Redis-backed worker model. This ensures tasks can be enqueued, managed, and processed efficiently by Cloud Run workers. See the application's configuration for details on queue integration and usage.

Operational intent:

  • Short-running tasks are pushed by the web app to Redis and executed by Cloud Run workers.
  • Other scheduled jobs are managed in Cloud Scheduler and run as k8s Jobs (they do not push to the worker queue).

Database migrations

If applicable to the service, database migrations are applied automatically when the application starts.

  • Migrations run as part of the main application service startup (the Cloud Run service), not in the worker pool.
  • You can track migration output and status in Cloud Run logs for the service (see the "Application Logs" links for each environment above).

Deployment Workflow

The project follows a GitFlow workflow with automatic deployments:

  1. Development: Commits to develop branch automatically trigger development environment deployment
  2. Staging: Merging develop to staging branch automatically triggers staging environment deployment
  3. Production: Merging staging to main branch automatically triggers production environment deployment

Each commit to these protected branches triggers the respective Cloud Build pipeline, which builds the Docker container and deploys it to the corresponding Cloud Run service.