Connecting BigQuery

Neuralift connects to BigQuery with a Google Cloud service account you provide, scoped to a single project. This page walks through creating that service account, adding the connection, and — when runs are published — provisioning the bucket, dataset, and BigLake connection that bring results back as queryable BigQuery tables (Receiving results in BigQuery).

How this connection works

A service account is Google Cloud’s identity for applications rather than people: you create one in your project, grant it read access to only the BigQuery datasets Neuralift needs, and hand Neuralift its JSON key. IAM on your side stays the source of truth. You can narrow the account’s roles or disable it at any time to sever the connection, and reads are attributable to that account in your audit logs when BigQuery Data Access audit logs are enabled in your project.

Before you begin

• You need the admin role in your Neuralift organization; see Roles & permissions.
• In Google Cloud you need permission to create service accounts and grant IAM roles in the project that holds your data.
• Neuralift does not accept any data with PII. Check Data requirements before connecting tables.

Create a service account

Create a dedicated service account for Neuralift, grant it read access, and export a JSON key:

PROJECT=<my-gcp-project>

# 1. Dedicated service account
gcloud iam service-accounts create neuralift-reader \
  --project=$PROJECT --display-name="Neuralift federation reader"

# 2. Roles: read the data, and use the BigQuery Storage Read API
gcloud projects add-iam-policy-binding $PROJECT \
  --member="serviceAccount:neuralift-reader@$PROJECT.iam.gserviceaccount.com" \
  --role=roles/bigquery.dataViewer
gcloud projects add-iam-policy-binding $PROJECT \
  --member="serviceAccount:neuralift-reader@$PROJECT.iam.gserviceaccount.com" \
  --role=roles/bigquery.readSessionUser

# 3. Export a JSON key to paste into Neuralift
gcloud iam service-accounts keys create neuralift-reader-key.json \
  --iam-account=neuralift-reader@$PROJECT.iam.gserviceaccount.com

A few notes:

• Prefer per-dataset scoping. Instead of project-wide roles/bigquery.dataViewer, grant the service account BigQuery Data Viewer on just the datasets Neuralift needs, via each dataset’s sharing/IAM settings. Keep roles/bigquery.readSessionUser at the project level.
• Organization policy on key creation. If your organization enforces iam.disableServiceAccountKeyCreation, step 3 is blocked; this integration needs a policy exception for the service account or project before it can proceed.
• VPC Service Controls. If your project sits inside a VPC-SC perimeter, the perimeter must allow Neuralift’s access to the BigQuery APIs; work with your Neuralift contact on the exception.
• Key handling and rotation. The key is used to provision the connection into Neuralift’s governed compute layer, where it’s encrypted; it is never stored in Neuralift’s own database. To rotate, create a new key and re-save the connection — it rotates in place.

Add the connection

1. In Neuralift, go to Settings → Data Connections and click Add connection.
2. Under Warehouse type, select BigQuery.
3. Enter a Connection name (lowercase letters, digits, and dashes; unique within your organization).
4. Enter your GCP project ID, for example <my-gcp-project>.
5. Paste the full contents of the key file into Service account key (JSON): the complete JSON document, starting with {"type": "service_account", ....
6. Click Save & connect.

The dialog closes and a Setting up … connection… notice appears. After a moment the new connection card appears with an Active badge (or Error with an explanation) and gains Browse tables and Test buttons.

Receiving results in BigQuery

Results are delivered into a GCS bucket you own and registered as queryable BigQuery tables through a BigLake connection, so each published run is a table your analysts query like any other. You create the bucket, dataset, and connection in your project; Neuralift writes the files and creates the tables. The delivery workflow around it is covered in Delivering results to your warehouse.

Two identities are involved — don’t conflate them:

• Neuralift’s delivery service account (one per organization; its email is shown on the delivery settings card) writes the Parquet files, runs BigQuery jobs, and creates the result tables.
• The BigLake connection’s Google-managed service agent (created by Google when you make the connection) is what BigQuery uses to read the delivered files at query time.

Each needs its own grants, made below.

1. Create the resources in your project

Everything you create, as one scriptable sequence. All three resources — bucket, dataset, and BigLake connection — must share the same region (or multi-region); mismatches only surface as errors at delivery time.

PROJECT=<my-gcp-project>
REGION=us-central1                # bucket, dataset, and connection must all use this
BUCKET=<my-neuralift-results>
DATASET=neuralift_results
CONNECTION=neuralift-delta

# 1. Enable the APIs
gcloud services enable storage.googleapis.com bigquery.googleapis.com \
  bigqueryconnection.googleapis.com --project=$PROJECT

# 2. Bucket the result files land in
gcloud storage buckets create gs://$BUCKET --project=$PROJECT \
  --location=$REGION --uniform-bucket-level-access

# 3. Dataset the result tables are created in
bq mk --dataset --location=$REGION $PROJECT:$DATASET

# 4. BigLake connection BigQuery reads the files through
bq mk --connection --connection_type=CLOUD_RESOURCE \
  --project_id=$PROJECT --location=$REGION $CONNECTION

# 5. Let the connection's service agent read the bucket
AGENT=$(bq show --connection --format=json $PROJECT.$REGION.$CONNECTION \
  | jq -r '.cloudResource.serviceAccountId')
gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
  --member="serviceAccount:$AGENT" --role=roles/storage.objectViewer

Datasets and connections are managed with the Google Cloud SDK’s bq command; there is no pure-gcloud surface for them.

2. Save the delivery configuration in Neuralift

Under Settings → Data Connections → Share your results, choose Cloud storage and enter your destination, gs://<bucket>/<prefix>. After saving, the settings card shows the email of Neuralift’s delivery service account and the exact grant commands for your resources — if a grant is missing, delivery errors name the precise command to run.

3. Grant Neuralift’s delivery service account access

NEURALIFT_SA=<delivery-service-account-email>   # shown on the delivery settings card

# Write and manage delivered files
gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
  --member="serviceAccount:$NEURALIFT_SA" --role=roles/storage.objectAdmin

# Run BigQuery jobs in your project
gcloud projects add-iam-policy-binding $PROJECT \
  --member="serviceAccount:$NEURALIFT_SA" --role=roles/bigquery.jobUser

# Create and update tables in the results dataset
bq add-iam-policy-binding \
  --member="serviceAccount:$NEURALIFT_SA" --role=roles/bigquery.dataEditor \
  $PROJECT:$DATASET

Then click Reset & re-provision on the delivery panel so Neuralift verifies the grants and finishes setup. Each published run your Neuralift team delivers appears as a table named sld_<run_id> in the dataset, backed by Parquet files under your bucket prefix.

Gotchas worth knowing up front

• Domain-restricted sharing. If your organization enforces iam.allowedPolicyMemberDomains, granting roles to Neuralift’s service account is blocked until you add a policy exception for its domain — this is the most common enterprise blocker.
• Older bq CLIs lack add-iam-policy-binding. Fall back to editing the dataset’s policy JSON: bq show --format=prettyjson $PROJECT:$DATASET > policy.json, add the binding, then bq update --source policy.json $PROJECT:$DATASET.
• Region colocation is mandatory (see above). If delivery fails with a DDL error mentioning locations, check that the bucket, dataset, and connection regions match.

Troubleshooting

• The connection lands on Error. The card shows an explanation. Common causes: the pasted key isn’t the complete JSON document, the key has been revoked, or the GCP project ID doesn’t match the project the service account belongs to.
• The connection is Active but expected tables are missing. The service account can only expose datasets it can read. Grant it access in Google Cloud, then use Sync tables in the Source tables sheet to re-discover them.
• Key creation fails in your project. Your organization policy blocks service-account key export (iam.disableServiceAccountKeyCreation); request an exception from your GCP administrators.
• BigQuery isn’t offered as a warehouse type. BigQuery connections and delivery are enabled per organization on request; see Release stages and contact support@neuralift.ai to request access.

Note. The service account key is used to provision the connection and is handled as described in Data handling & security.

Next steps

• Sync source tables from your new connection.
• Review data requirements so your tables are ready for segmentation.
• When runs are published, see Delivering results to your warehouse for the delivery workflow.