TAP_SCHEMA metadata management#

An IVOA-compliant TAP server requires an underlying TAP_SCHEMA database that holds a specification of its schema. This table is used internally by the TAP server and can also be queried directly by the user to understand the schema of the data served by that TAP server.

Repertoire can manage TAP_SCHEMA metadata for TAP servers that use either Cloud SQL or external PostgreSQL databases. When TAP_SCHEMA management is enabled for a TAP service, Repertoire will automatically populate and update the schema metadata whenever its Helm chart is deployed (in Phalanx, usually via an Argo CD sync).

For TAP applications using containerized (in-cluster) databases, see Containerized databases.

Overview#

TAP_SCHEMA is a set of database tables that describe the schemas, tables, and columns available through a TAP service. Clients use this metadata to discover what data is queryable, either via ADQL queries or using the VOSI /tables endpoint on the TAP server.

Repertoire populates TAP_SCHEMA by:

Downloading schema definitions from sdm_schemas, or another configurable source that produces the same artifacts.
Parsing the YAML schema files using Felis.
Loading the schema metadata into a staging PostgreSQL schema.
Validating the staged data.
Performing an atomic swap to make the new metadata live.

This process runs automatically as a Helm pre-install/pre-upgrade hook during Argo CD syncs of Repertoire.

If the Cloud SQL proxy is needed (cloudsql.enabled in Repertoire’s Phalanx values), it runs as an init container in the job pod to provide connectivity.

Configuration#

TAP_SCHEMA management is configured in the Repertoire application’s Helm values in Phalanx, specifically in applications/repertoire/values.yaml and environment-specific overrides.

Global settings#

This configuration applies to all TAP servers unless overridden per-server.

config.tap.schemaVersion

The default sdm_schemas version to use. This is a tag from the sdm_schemas repository.

config.tap.schemaSourceTemplate

URL template for downloading the sdm_schemas archive. The literal string {version} is replaced with the schema version as set by config.tap.schemaVersion or in an override specific to a given TAP server. Default: https://github.com/lsst/sdm_schemas/archive/refs/tags/{version}.tar.gz

Supported URL schemes: https, http, and gs (Google Cloud Storage).

To use the GCS artifacts published by the sdm_schemas workflow, use one of the following templates:

Tagged release: gs://rubin-sdm-schemas-artifacts/releases/{version}/schemas.tar.gz with schemaVersion set to the tag name (e.g. w.2025.01.1).
Ticket branch: gs://rubin-sdm-schemas-artifacts/{version}/schemas.tar.gz with schemaVersion set to the full branch name (e.g. tickets/DM-54182).

The template differs between the two cases because the sdm_schemas build workflow stores tagged releases under releases/tag/ in GCS, while for ticket branches the path is tickets/branch/.

Per-server settings#

Each TAP server is configured under config.tap.servers.<name>, where <name> matches the Phalanx application name (e.g., tap, ssotap, livetap, etc.).

config.tap.servers.<name>.enabled

Whether Repertoire should manage updating the TAP_SCHEMA for this server. Set to false in environments where the TAP server doesn’t exist or uses containerized databases. Default: false.

config.tap.servers.<name>.schemas

List of schema names to load. These correspond to YAML file names (without the .yaml extension) in the sdm_schemas package. The order of this list determines the display order in TAP clients via the tap_schema_index parameter passed to Felis.

Example:

schemas:
  - "dp1"
  - "ivoa_obscore"
  - "dp02_dc2"

In this example, dp1 will appear first (index 1), ivoa_obscore second (index 2), and dp02_dc2 third (index 3).

config.tap.servers.<name>.schemaVersion

Override the global schemaVersion for this specific server. If not set, the global config.tap.schemaVersion is used.

config.tap.servers.<name>.databaseUrl

PostgreSQL connection URL for the TAP_SCHEMA database. Format: postgresql://user@host:port/database

When using the CloudSQL backend, the host and port should be set to 127.0.0.1:5432 to use the built-in Cloud SQL Auth Proxy.

config.tap.servers.<name>.databasePasswordKey

Key name in the Repertoire Vault secret that contains the database password.

Full example#

applications/repertoire/values.yaml#

config:
  tap:
    schemaVersion: "w.2026.01"
    schemaSourceTemplate: >-
      https://github.com/lsst/sdm_schemas/archive/refs/tags/{version}.tar.gz
    servers:
      tap:
        enabled: false
        schemas:
          - "dp1"
          - "ivoa_obscore"
          - "dp02_dc2"
        databaseUrl: "postgresql://tap@127.0.0.1:5432/tap"
        databasePasswordKey: "tap-database-password"
      ssotap:
        enabled: false
        schemas:
          - "dp03_10yr"
          - "dp03_1yr"
        databaseUrl: "postgresql://ssotap@127.0.0.1:5432/ssotap"
        databasePasswordKey: "ssotap-database-password"

applications/repertoire/values-idfint.yaml#

config:
  tap:
    servers:
      tap:
        enabled: true
      ssotap:
        enabled: true

cloudsql:
  enabled: true
  instanceConnectionName: "project:region:instance"
  serviceAccount: "repertoire@project.iam.gserviceaccount.com"

Common tasks#

Updating the sdm_schemas version#

To update the schema definitions to a new sdm_schemas release:

Change config.tap.schemaVersion in applications/repertoire/values.yaml to the new version tag for sdm_schemas.
If the cadc-tap chart’s config.datalinkPayloadUrl references a specific sdm_schemas version, update that as well in charts/cadc-tap/values.yaml.
Commit and push the changes, then sync the Repertoire application via ArgoCD. A helm hook will run for each enabled TAP service, and will automatically download the new schema version and update TAP_SCHEMA.

Adding or removing schemas from a TAP server#

To change which schemas are served by a TAP application:

Edit the schemas list under config.tap.servers.<name> in applications/repertoire/values.yaml.
Add or remove schemas. Added schema names must match YAML file names in the sdm_schemas package (without the .yaml extension).
Sync the Repertoire application. The Helm hook will reload all schemas for that server.

Changing schema display order#

The order of schemas in the schemas list directly controls the tap_schema_index used by TAP clients such as Firefly to order schemas in their UI.

# dp1 will appear first, then dp02_dc2, then ivoa_obscore
schemas:
  - "dp1"
  - "dp02_dc2"
  - "ivoa_obscore"

Therefore, to change the order, fix the order in application/repertoire/values.yaml and then sync the Repertoire application.

Testing schemas from a ticket branch#

During development on sdm_schemas, you can test unreleased schema changes in an environment by pointing Repertoire at the GCS artifacts published from a ticket branch.

Push your changes to a tickets/* branch in the sdm_schemas repository. The build workflow will automatically upload the schema to GCS at gs://rubin-sdm-schemas-artifacts/tickets/<branch>/schemas.tar.gz.

In the appropriate environment values file (e.g. applications/repertoire/values-idfint.yaml), override the TAP schema settings:

config:
  tap:
    schemaVersion: "tickets/DM-XXXXX"
    schemaSourceTemplate: >-
      gs://rubin-sdm-schemas-artifacts/{version}/schemas.tar.gz

Sync Repertoire via Argo CD. The Helm hook will download the schema archive from the ticket branch path in GCS and update TAP_SCHEMA.
Once the ticket is merged and a release tag is pushed, revert to the standard configuration using the tagged release and the GCS release template.

Adding a new TAP server to Repertoire management#

To have Repertoire manage TAP_SCHEMA for a new TAP application:

Add a new entry under config.tap.servers in applications/repertoire/values.yaml:

config:
  tap:
    servers:
      newtap:
        enabled: false
        schemas:
          - "my_schema"
        databaseUrl: "postgresql://newtap@127.0.0.1:5432/newtap"
        databasePasswordKey: "newtap-database-password"

Add the corresponding secret to applications/repertoire/secrets.yaml:

newtap-database-password:
  description: >-
    Database password for the newtap service, used by repertoire to manage
    the newtap TAP_SCHEMA.
  if: config.tap.servers.newtap.enabled
  copy:
    application: newtap
    key: tap-schema-password

Enable the server in the appropriate environment values file (e.g., applications/repertoire/values-idfint.yaml):
```
config:
  tap:
    servers:
      newtap:
        enabled: true
```
Ensure the TAP application itself has tap-schema-password in its Vault secrets and that the TAP_SCHEMA database type is set to cloudsql or external (not containerized).

Secrets#

Repertoire requires database credentials to connect to each TAP server’s TAP_SCHEMA database. Database passwords are copied from the corresponding TAP application’s Vault secrets using the copy directive in applications/repertoire/secrets.yaml.

This means you only need to manage the password in one place (the TAP application’s Vault path). Repertoire will automatically have access to it after secrets are synced for an environment with phaalnx secrets sync. See the Phalanx documentation for more details.

The password key name for each server is configured via databasePasswordKey in the server configuration.

Containerized databases#

When a TAP application uses the containerized (in-cluster) database backend, Repertoire does not manage its TAP_SCHEMA metadata. For this setup, the schema metadata is embedded directly in the MySQL Docker image that runs as the TAP_SCHEMA database pod.

Warning

This schema management approach is deprecated and should not be used for any new TAP server deployment. It will be removed entirely as soon as all existing TAP servers have switched to the new schema management approach.

These images are configured in the TAP application’s Helm values:

cadc-tap:
  tapSchema:
    type: "containerized"
    image:
      repository: "lsstsqre/tap-schema-usdf-prod-tap"
      tag: "w.2026.01"

To update TAP_SCHEMA metadata for a containerized deployment:

Build a new Docker image containing the updated schema metadata.
Push the image to the container registry.
Update tapSchema.image.tag in the environment-specific TAP values file.
Sync the TAP application via Argo CD.

For information on setting up TAP database infrastructure, see the Phalanx TAP database configuration guide.

Next steps#

Services: Service rules
Data collections: Datasets
HiPS service: HiPS service discovery configuration
InfluxDB databases: InfluxDB databases