Store events from Twilio Segment
Author: Adam Tworkiewicz
Last updated: January 18th, 2022
Overview
This solution listens to events from Twilio Segment and writes data to a Teradata Vantage instance. The example uses GCP but it can be translated into any cloud platform.
Architecture
In this solution, Twilio Segment writes raw event data to GCP Pub/Sub. Pub/Sub forwards events to a Cloud Run application. The Cloud Run app writes data to a Teradata Vantage database. It’s a serverless solution that doesn’t require allocation or management of any VM’s.

Deployment
Prerequsites
-
A GCP account. If you don’t have an account, you can create one at https://console.cloud.google.com/.
-
gcloud
installed. See https://cloud.google.com/sdk/docs/install. -
A Vantage Express instance that GCP Cloud run can talk to. See Run Vantage Express on GCP for detailed instructions.
Build and deploy
-
Clone the sample repository:
git clone git@github.com:Teradata/segment-integration-tutorial.git
-
The repo contains
segment.sql
file that sets up the database. the script on your Vantage db using your favorite SQL IDE, [Teradata Studio](https://downloads.teradata.com/download/tools/teradata-studio) or command line tool calledbteq
(download for Windows, Linux, macOS). The SQL script will create a new database calledSegment
and a set of tables to store Segment events. -
Set the default project and region:
gcloud config set project <PROJECT_ID> gcloud config set compute/region <REGION>
-
Retrieve the project id and the number. We will need it in subsequent steps:
export PROJECT_ID=$(gcloud config get-value project) export PROJECT_NUMBER=$(gcloud projects list \ --filter="$(gcloud config get-value project)" \ --format="value(PROJECT_NUMBER)")
-
Enable required GCP services:
gcloud services enable cloudbuild.googleapis.com containerregistry.googleapis.com run.googleapis.com secretmanager.googleapis.com pubsub.googleapis.com
-
Build the application:
gcloud builds submit --tag gcr.io/$PROJECT_ID/segment-listener
-
Define an API key that you will share with Segment. Store the API key in GCP Secret Manager:
gcloud secrets create VANTAGE_USER_SECRET echo -n 'dbc' > /tmp/vantage_user.txt gcloud secrets versions add VANTAGE_USER_SECRET --data-file=/tmp/vantage_user.txt gcloud secrets create VANTAGE_PASSWORD_SECRET echo -n 'dbc' > /tmp/vantage_password.txt gcloud secrets versions add VANTAGE_PASSWORD_SECRET --data-file=/tmp/vantage_password.txt
-
The application that write Segment data to Vantage will use Cloud Run. We first need to allow Cloud Run to access secrets:
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member=serviceAccount:$PROJECT_NUMBER-compute@developer.gserviceaccount.com \ --role=roles/secretmanager.secretAccessor
-
Deploy the app to Cloud Run (replace
<VANTAGE_HOST>
with the hostname or IP of your Teradata Vantage database). The second export statement saves the service url as we need it for subsequent commands:gcloud run deploy --image gcr.io/$PROJECT_ID/segment-listener segment-listener \ --region $(gcloud config get-value compute/region) \ --update-env-vars VANTAGE_HOST=35.239.251.1 \ --update-secrets 'VANTAGE_USER=VANTAGE_USER_SECRET:1, VANTAGE_PASSWORD=VANTAGE_PASSWORD_SECRET:1' \ --no-allow-unauthenticated export SERVICE_URL=$(gcloud run services describe segment-listener --platform managed --region $(gcloud config get-value compute/region) --format 'value(status.url)')
-
Create a Pub/Sub topic that will receive events from Segment:
gcloud pubsub topics create segment-events
-
Create a service account that will be used by Pub/Sub to invoke the Cloud Run app:
gcloud iam service-accounts create cloud-run-pubsub-invoker \ --display-name "Cloud Run Pub/Sub Invoker"
-
Give the service account permission to invoke Cloud Run:
gcloud run services add-iam-policy-binding segment-listener \ --region $(gcloud config get-value compute/region) \ --member=serviceAccount:cloud-run-pubsub-invoker@$PROJECT_ID.iam.gserviceaccount.com \ --role=roles/run.invoker
-
Allow Pub/Sub to create authentication tokens in your project:
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member=serviceAccount:service-$PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com \ --role=roles/iam.serviceAccountTokenCreator
-
Create a Pub/Sub subscription with the service account:
gcloud pubsub subscriptions create segment-events-cloudrun-subscription --topic projects/$PROJECT_ID/topics/segment-events \ --push-endpoint=$SERVICE_URL \ --push-auth-service-account=cloud-run-pubsub-invoker@$PROJECT_ID.iam.gserviceaccount.com \ --max-retry-delay 600 \ --min-retry-delay 30
-
Allow Segment to publish to your topic. To do that, assign
pubsub@segment-integrations.iam.gserviceaccount.com
rolePub/Sub Publisher
in your project at https://console.cloud.google.com/cloudpubsub/topic/list. See Segment manual for details. -
Configure your GCP Pub/Sub a destination in Segment. Use the full topic
projects/<PROJECT_ID>/topics/segment-events
and map all Segment event types (using*
character) to the topic.
Try it out
-
Use Segment’s Event Tester functionality to send a sample payload to the topic. Verify that the sample data has been stored in Vantage.
Limitations
-
The example shows how to deploy the app in a single region. In many cases, this setup doesn’t guarantee enough uptime. The Cloud Run app should be deployed in more than one region behind a Global Load Balancer.
Summary
This how-to demonstrates how to send Segment events to Teradata Vantage. The configuration forwards events from Segment to GCP Pub/Sub and then on to a Cloud Run application. The application writes data to Teradata Vantage.