Store events from Twilio Segment

Author: Adam Tworkiewicz
Last updated: January 18th, 2022

Overview

This solution listens to events from Twilio Segment and writes data to a Teradata Vantage instance. The example uses Google Cloud but it can be translated into any cloud platform.

Architecture

In this solution, Twilio Segment writes raw event data to Google Cloud Pub/Sub. Pub/Sub forwards events to a Cloud Run application. The Cloud Run app writes data to a Teradata Vantage database. It’s a serverless solution that doesn’t require allocation or management of any VM’s.

Deployment

Prerequisites

A Google Cloud account. If you don’t have an account, you can create one at https://console.cloud.google.com/.
gcloud installed. See https://cloud.google.com/sdk/docs/install.

A Teradata Vantage instance that Google Cloud Run can talk to.

If you need a new instance of Vantage, you can install a free version called Vantage Express in the cloud on Google Cloud, Azure, and AWS. You can also run Vantage Express on your local machine using VMware, VirtualBox, or UTM.

Build and deploy

Clone the sample repository:

git clone git@github.com:Teradata/segment-integration-tutorial.git

The repo contains segment.sql file that sets up the database. the script on your Vantage db using your favorite SQL IDE, [Teradata Studio](https://downloads.teradata.com/download/tools/teradata-studio) or command line tool called bteq (download for Windows, Linux, macOS). The SQL script will create a new database called Segment and a set of tables to store Segment events.

Set the default project and region:

gcloud config set project <PROJECT_ID>
gcloud config set compute/region <REGION>

Retrieve the project id and the number. We will need it in subsequent steps:

export PROJECT_ID=$(gcloud config get-value project)

export PROJECT_NUMBER=$(gcloud projects list \
  --filter="$(gcloud config get-value project)" \
  --format="value(PROJECT_NUMBER)")

Enable required Google Cloud services:

gcloud services enable cloudbuild.googleapis.com containerregistry.googleapis.com run.googleapis.com secretmanager.googleapis.com pubsub.googleapis.com

Build the application:

gcloud builds submit --tag gcr.io/$PROJECT_ID/segment-listener

Define an API key that you will share with Segment. Store the API key in Google Cloud Secret Manager:

gcloud secrets create VANTAGE_USER_SECRET
echo -n 'dbc' > /tmp/vantage_user.txt
gcloud secrets versions add VANTAGE_USER_SECRET --data-file=/tmp/vantage_user.txt

gcloud secrets create VANTAGE_PASSWORD_SECRET
echo -n 'dbc' > /tmp/vantage_password.txt
gcloud secrets versions add VANTAGE_PASSWORD_SECRET --data-file=/tmp/vantage_password.txt

The application that write Segment data to Vantage will use Cloud Run. We first need to allow Cloud Run to access secrets:

gcloud projects add-iam-policy-binding $PROJECT_ID \
     --member=serviceAccount:$PROJECT_NUMBER-compute@developer.gserviceaccount.com \
     --role=roles/secretmanager.secretAccessor

Deploy the app to Cloud Run (replace <VANTAGE_HOST> with the hostname or IP of your Teradata Vantage database). The second export statement saves the service url as we need it for subsequent commands:

gcloud run deploy --image gcr.io/$PROJECT_ID/segment-listener segment-listener \
  --region $(gcloud config get-value compute/region) \
  --update-env-vars VANTAGE_HOST=35.239.251.1 \
  --update-secrets 'VANTAGE_USER=VANTAGE_USER_SECRET:1, VANTAGE_PASSWORD=VANTAGE_PASSWORD_SECRET:1' \
  --no-allow-unauthenticated

export SERVICE_URL=$(gcloud run services describe segment-listener --platform managed --region $(gcloud config get-value compute/region) --format 'value(status.url)')

Create a Pub/Sub topic that will receive events from Segment:
```
gcloud pubsub topics create segment-events
```

Create a service account that will be used by Pub/Sub to invoke the Cloud Run app:

gcloud iam service-accounts create cloud-run-pubsub-invoker \
     --display-name "Cloud Run Pub/Sub Invoker"

Give the service account permission to invoke Cloud Run:

gcloud run services add-iam-policy-binding segment-listener \
  --region $(gcloud config get-value compute/region) \
  --member=serviceAccount:cloud-run-pubsub-invoker@$PROJECT_ID.iam.gserviceaccount.com \
  --role=roles/run.invoker

Allow Pub/Sub to create authentication tokens in your project:

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member=serviceAccount:service-$PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com \
  --role=roles/iam.serviceAccountTokenCreator

Create a Pub/Sub subscription with the service account:

gcloud pubsub subscriptions create segment-events-cloudrun-subscription --topic projects/$PROJECT_ID/topics/segment-events \
   --push-endpoint=$SERVICE_URL \
   --push-auth-service-account=cloud-run-pubsub-invoker@$PROJECT_ID.iam.gserviceaccount.com \
   --max-retry-delay 600 \
   --min-retry-delay 30

Allow Segment to publish to your topic. To do that, assign pubsub@segment-integrations.iam.gserviceaccount.com role Pub/Sub Publisher in your project at https://console.cloud.google.com/cloudpubsub/topic/list. See Segment manual for details.
Configure your Google Cloud Pub/Sub a destination in Segment. Use the full topic projects/<PROJECT_ID>/topics/segment-events and map all Segment event types (using * character) to the topic.

Try it out

Use Segment’s Event Tester functionality to send a sample payload to the topic. Verify that the sample data has been stored in Vantage.

Limitations

The example shows how to deploy the app in a single region. In many cases, this setup doesn’t guarantee enough uptime. The Cloud Run app should be deployed in more than one region behind a Global Load Balancer.

Summary

This how-to demonstrates how to send Segment events to Teradata Vantage. The configuration forwards events from Segment to Google Cloud Pub/Sub and then on to a Cloud Run application. The application writes data to Teradata Vantage.