aqua
Go back

Managing Secrets in QA Without Compromising Security

Every QA workflow needs credentials. Login passwords, API keys, admin tokens, MFA secrets — you can’t test a real application without them. But credentials in QA have a way of ending up in places they shouldn’t: pasted into chat messages, hardcoded in test scripts, logged in CI output, or — in the AI agent era — sent to a language model that remembers everything.

In this post, we’ll look at how aqua handles secrets so they never leave your machine in plain text, even when an AI agent is driving the tests.

The Problem: Secrets and AI Agents

Traditional QA tools assume a human is running the tests. The human knows the password, types it in, and that’s that. But when an AI coding agent creates and executes QA plans, a new question arises: does the agent need to see the actual secret values?

With aqua, the answer is no. Secrets are resolved locally on your machine by the aqua CLI process, not by the AI agent. The agent tells aqua which secret to use (via a template variable like {{test_user_password}}), and aqua resolves it from your configured secret provider. The agent never sees the raw value.

This is a fundamental design choice, not a feature bolted on afterward. Let’s look at how it works in practice.

Environment Files: Where Secrets Live

Secrets are configured in environment files at .aqua/environments/<name>.json. Each environment maps variable names to secret entries with a type that tells aqua where to fetch the value from.

{
  "notes": "Staging environment with 1Password and AWS secrets",
  "variables": {
    "api_base_url": "https://staging-api.example.com",
    "web_base_url": "https://staging.example.com"
  },
  "secrets": {
    "test_user_password": {
      "type": "op",
      "value": "op://Development/staging-app/password"
    },
    "admin_api_key": {
      "type": "aws_sm",
      "value": "staging/admin-api-key"
    },
    "mfa_secret": {
      "type": "env",
      "value": "STAGING_MFA_SECRET"
    }
  }
}

The variables section holds non-sensitive values like URLs. The secrets section holds sensitive values — each with a type that determines how it’s resolved.

Environment files live in your repository (or not — it’s up to you) and are never sent to aqua’s server. They’re read locally by the CLI when you execute a plan.

Six Ways to Provide Secrets

aqua supports six secret types, from simple to enterprise-grade:

literal — Direct Values

{
  "test_user_password": {
    "type": "literal",
    "value": "dev-password-123"
  }
}

The value is used as-is. This is fine for local development with throwaway credentials, but avoid it for anything sensitive. The password is right there in the file.

env — Environment Variables

{
  "ci_token": {
    "type": "env",
    "value": "CI_DEPLOY_TOKEN"
  }
}

Reads from the OS environment variable CI_DEPLOY_TOKEN. This is the natural choice for CI/CD pipelines where secrets are injected by the runner (GitHub Actions secrets, GitLab CI variables, etc.).

op — 1Password CLI

{
  "test_user_password": {
    "type": "op",
    "value": "op://Development/staging-app/password"
  }
}

Fetches the value from 1Password using the op CLI. The reference follows 1Password’s URI format: op://vault/item/field. Requires the 1Password CLI to be installed and authenticated on your machine.

This is a great choice for local development — your team likely already uses 1Password, and the CLI integrates with biometric unlock. No extra tooling to set up.

aws_sm — AWS Secrets Manager

{
  "db_password": {
    "type": "aws_sm",
    "value": "production/db-credentials",
    "json_key": "password"
  }
}

Fetches from AWS Secrets Manager using the AWS CLI. If the secret value is JSON, json_key extracts a specific field. You can also specify region and profile per entry, or set defaults in the secret_providers section.

gcp_sm — GCP Secret Manager

{
  "api_key": {
    "type": "gcp_sm",
    "value": "staging-api-key",
    "project": "my-project-123",
    "version": "latest"
  }
}

Fetches from GCP Secret Manager using gcloud. Like AWS, supports json_key for JSON secrets and per-entry or provider-level project defaults.

hcv — HashiCorp Vault

{
  "signing_key": {
    "type": "hcv",
    "value": "myapp/signing",
    "field": "private_key",
    "mount": "secret"
  }
}

Fetches from HashiCorp Vault using the vault CLI. Supports both KV v1 and v2 engines. field extracts a specific field from the secret data, and mount specifies the secrets engine mount point.

Provider-Level Defaults

When you use multiple secrets from the same provider, setting defaults avoids repetition:

{
  "secrets": {
    "db_password": {
      "type": "aws_sm",
      "value": "staging/db-credentials",
      "json_key": "password"
    },
    "api_key": {
      "type": "aws_sm",
      "value": "staging/api-key"
    }
  },
  "secret_providers": {
    "aws_sm": {
      "region": "ap-northeast-1",
      "profile": "staging"
    }
  }
}

Both AWS secrets inherit the region and profile from secret_providers. Individual entries can still override these if needed.

Lazy Resolution: Only Fetch What You Need

aqua doesn’t resolve all secrets in your environment file upfront. Before execution, it scans the QA plan for template variable references ({{variable_name}}), then resolves only the secrets that are actually used.

This matters for two reasons:

  1. Fewer API calls. If your environment file has 20 secrets but the plan only uses 3, only 3 calls are made to your secret providers.
  2. Fewer failure points. A secret that requires AWS credentials you haven’t configured won’t cause an error if it’s not referenced in the plan you’re running.

If a plan references a secret that can’t be resolved (wrong 1Password path, expired AWS token, etc.), you get a clear error before execution starts — with a suggestion for how to fix it (e.g., “run op signin” or “run aws configure”).

Multi-Layer Masking

Secret resolution is only half the story. The other half is making sure secrets don’t appear in execution results — the data that gets sent to aqua’s server and shown in the dashboard.

aqua applies five masking rules before any data leaves your machine:

1. Secret value masking — Every resolved secret value is tracked. Before sending HTTP requests, responses, and DOM snapshots to the server, aqua scans all string content for these values and replaces them with ***. This is the safety net — if a secret appears anywhere in any response, it gets caught.

2. Environment masking — In execution results, the resolved environment dictionary replaces all secret values with ***. Anyone viewing the results sees which variables were used, but not their values.

3. Authorization header masking — The Authorization header in HTTP requests is always masked, regardless of whether it contains a tracked secret.

4. Set-Cookie header masking — Set-Cookie headers in HTTP responses are masked to prevent session tokens from leaking into results.

5. DOM password field masking — In browser DOM snapshots, <input type="password"> elements have their value attributes masked.

These rules are layered intentionally. The secret value scan catches the general case, while the specific rules handle known sensitive patterns that might not match a tracked secret (e.g., a session token returned by the server that wasn’t in your environment file).

The masking happens in the CLI process on your machine — the server only ever receives masked data.

Working with MFA and TOTP

Multi-factor authentication is common in staging and production environments. aqua supports TOTP (Time-based One-Time Password) natively with the {{totp:variable_name}} syntax.

Store your TOTP secret in the environment file using any secret type:

{
  "secrets": {
    "mfa_secret": {
      "type": "op",
      "value": "op://Development/staging-app/one-time-password"
    }
  }
}

Then reference it in a browser step:

{
  "step_key": "enter_mfa_code",
  "action": "browser",
  "config": {
    "steps": [
      { "wait_for_selector": "input[name='totp_code']" },
      { "type": { "selector": "input[name='totp_code']", "text": "{{totp:mfa_secret}}" } },
      { "click": "button[type='submit']" }
    ]
  }
}

At execution time, aqua resolves the MFA secret from 1Password, computes a fresh 6-digit TOTP code, and types it into the form. The underlying secret and the generated code are both masked in results.

The TOTP generator supports both raw Base32 secrets and otpauth:// URIs with custom parameters (algorithm, digit count, period).

Corporate Proxies

If your staging or production environment sits behind a corporate proxy, the environment file supports proxy configuration with credentials:

{
  "proxy": {
    "server": "http://proxy.corp.com:3128",
    "bypass": "localhost,.internal.com",
    "username": { "type": "env", "value": "PROXY_USER" },
    "password": { "type": "env", "value": "PROXY_PASSWORD" }
  }
}

Proxy credentials use the same secret types as regular secrets — you can pull them from 1Password, environment variables, or any other provider. They’re resolved and masked using the same pipeline.

Patterns for Different Environments

Here’s how a typical team might structure their environments:

Local Development

{
  "notes": "Local dev — throwaway credentials, no external secret managers needed",
  "variables": {
    "api_base_url": "http://localhost:8000",
    "web_base_url": "http://localhost:3000"
  },
  "secrets": {
    "test_user_email": { "type": "literal", "value": "dev@example.com" },
    "test_user_password": { "type": "literal", "value": "devpassword" }
  }
}

Local credentials are throwaway — literal is fine here. The database gets wiped regularly anyway.

Staging

{
  "notes": "Staging — shared credentials via 1Password",
  "variables": {
    "api_base_url": "https://staging-api.example.com",
    "web_base_url": "https://staging.example.com"
  },
  "secrets": {
    "test_user_email": { "type": "op", "value": "op://QA/staging-app/username" },
    "test_user_password": { "type": "op", "value": "op://QA/staging-app/password" },
    "admin_email": { "type": "op", "value": "op://QA/staging-app-admin/username" },
    "admin_password": { "type": "op", "value": "op://QA/staging-app-admin/password" },
    "mfa_secret": { "type": "op", "value": "op://QA/staging-app-admin/one-time-password" }
  }
}

1Password is a natural fit for staging — credentials are shared across the team via a vault, and each developer resolves them locally using their own 1Password session.

CI/CD

{
  "notes": "CI — secrets injected by GitHub Actions",
  "variables": {
    "api_base_url": "https://staging-api.example.com",
    "web_base_url": "https://staging.example.com"
  },
  "secrets": {
    "test_user_email": { "type": "env", "value": "QA_USER_EMAIL" },
    "test_user_password": { "type": "env", "value": "QA_USER_PASSWORD" }
  }
}

In CI, secrets come from the runner’s environment. GitHub Actions, GitLab CI, and similar platforms inject secrets as environment variables — the env type picks them up directly.

Production

{
  "notes": "Production — secrets from AWS Secrets Manager, read-only test account",
  "variables": {
    "api_base_url": "https://api.example.com",
    "web_base_url": "https://app.example.com"
  },
  "secrets": {
    "test_user_email": { "type": "aws_sm", "value": "prod/qa-readonly-account", "json_key": "email" },
    "test_user_password": { "type": "aws_sm", "value": "prod/qa-readonly-account", "json_key": "password" }
  },
  "secret_providers": {
    "aws_sm": { "region": "us-east-1", "profile": "production-readonly" }
  }
}

Production secrets should come from your organization’s secrets infrastructure — AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault. The IAM profile is scoped to read-only access.

The Security Model at a Glance

To summarize how secrets flow through aqua:

  1. You configure secret references in .aqua/environments/<name>.json
  2. The CLI resolves only the secrets referenced by the plan, locally on your machine
  3. Template variables ({{secret_name}}) are expanded with real values during execution
  4. Multi-layer masking replaces all secret values with *** in execution results
  5. Masked results are sent to the server for storage and dashboard display
  6. The AI agent sees only template variable names and masked results — never raw values

At no point do raw secret values leave your machine. The server stores only masked data. The AI agent works only with variable references. This is true whether you run aqua from the CLI, through an AI coding agent, or in CI/CD.

What’s Next

With secrets handled securely, the natural next question is: how do you run the same plan across local, staging, and production without maintaining separate configurations? In the next post, we’ll cover multi-environment QA — reusing plans across environments, handling environment-specific data, and building a promotion workflow from local to production.

Share this post on:
Share this post on X
Older Post
Writing Effective QA Plans: Tips and Patterns