Pass-account integration between SaaS platforms the usage of Amazon AppFlow

Imposing an efficient information sharing technique that satisfies compliance and regulatory necessities is advanced. Consumers regularly wish to proportion information between disparate instrument as a carrier (SaaS) platforms inside of their group or throughout organizations. On many events, they wish to follow industry good judgment to the knowledge gained from the supply SaaS platform prior to pushing it to the objective SaaS platform.

Letâs take an instance. AnyCompanyâs advertising group hosted an occasion on the Anaheim Conference Middle, CA. The promoting group created leads in keeping with the development in Adobe Marketo. An automatic procedure downloaded the leads from Marketo within the advertising AWS account. Those leads are then driven to the gross sales AWS account. A industry procedure choices up the ones leads, filters them in keeping with a âDo No longer Nameâ standards, and creates entries within the Salesforce gadget. Now, the gross sales group can pursue the ones leads and proceed to trace the alternatives in Salesforce.

On this submit, we display how one can proportion your information throughout SaaS platforms in a cross-account construction the usage of absolutely controlled, low-code AWS services and products akin to Amazon AppFlow, Amazon EventBridge, AWS Step Purposes, and AWS Glue.

Resolution review

Bearing in mind our instance of AnyCompany, letâs take a look at the knowledge circulate. AnyCompanyâs Marketo example is built-in with the manufacturer AWS account. Because the leads from Marketo land within the manufacturer AWS account, theyâre driven to the patron AWS account, which is built-in to Salesforce. Industry good judgment is implemented to the leads information within the person AWS account, after which the curated information is loaded into Salesforce.

We now have used a serverless structure to put into effect this use case. The next AWS services and products are used for information ingestion, processing, and cargo:

Amazon AppFlow is an absolutely controlled integration carrier that lets you securely switch information between SaaS packages like Salesforce, SAP, Marketo, Slack, and ServiceNow, and AWS services and products like Amazon S3 and Amazon Redshift, in only some clicks. With AppFlow, you’ll be able to run information flows at just about any scale on the frequency you selectâon a time table, according to a industry occasion, or on call for. You’ll be able to configure information transformation functions like filtering and validation to generate wealthy, ready-to-use information as a part of the circulate itself, with out further steps. Amazon AppFlow is used to obtain leads information from Marketo and add the curated leads information into Salesforce.
Amazon EventBridge is a serverless occasion bus that permits you to obtain, filter out, turn out to be, path, and ship occasions. EventBridge is used to trace the occasions like receiving the leads information within the manufacturer or person AWS accounts after which triggering a workflow.
AWS Step Purposes is a visible workflow carrier that is helping builders use AWS services and products to construct allotted packages, automate processes, orchestrate microservices, and create information and mechanical device studying (ML) pipelines. Step Purposes is used to orchestrate the knowledge processing.
AWS Glue is a serverless information preparation carrier that makes it simple to run extract, turn out to be, and cargo (ETL) jobs. An AWS Glue task encapsulates a script that reads, processes, after which writes information to a brand new schema. This answer makes use of Python 3.6 AWS Glue jobs for information filtration and processing.
Amazon Easy Garage Carrier (Amazon S3) is an object garage carrier providing industry-leading scalability, information availability, safety, and function. Amazon S3 is used to retailer the leads information.

Letâs overview the structure intimately. The next diagram displays a visible illustration of ways this integration works.

The next steps define the method for moving and processing leads information the usage of Amazon AppFlow, Amazon S3, EventBridge, Step Purposes, AWS Glue, and Salesforce:

Amazon AppFlow runs on a day-to-day time table and retrieves any new leads created throughout the ultimate 24 hours (incremental adjustments) from Marketo.
The leads are stored as Parquet layout recordsdata in an S3 bucket within the manufacturer account.
When the day-to-day circulate is entire, Amazon AppFlow emits occasions to EventBridge.
EventBridge triggers Step Purposes.
Step Purposes copies the Parquet layout recordsdata containing the leads from the manufacturer accountâs S3 bucket to the patron accountâs S3 bucket.
Upon a a hit report switch, Step Purposes publishes an occasion within the person accountâs EventBridge.
An EventBridge rule intercepts this occasion and triggers Step Purposes within the person account.
Step Purposes calls an AWS Glue crawler, which scans the leads Parquet recordsdata and creates a desk within the AWS Glue Knowledge Catalog.
The AWS Glue task is named, which selects information with the Do No longer Name box set to false from the leads recordsdata, and creates a brand new set of curated Parquet recordsdata. We now have used an AWS Glue task for the ETL pipeline to show off how you’ll be able to use purpose-built analytics carrier for advanced ETL wishes. Then again, for easy filtering necessities like Do No longer Name, you’ll be able to use the present filtering function of Amazon AppFlow.
Step Purposes then calls Amazon AppFlow.
In spite of everything, Amazon AppFlow populates the Salesforce leads in keeping with the knowledge within the curated Parquet recordsdata.

We now have supplied artifacts on this submit to deploy the AWS services and products on your account and check out out the answer.

Must haves

To observe the deployment walkthrough, you want two AWS accounts, one for the manufacturer and different for the patron. Use us-east-1 or us-west-2 as your AWS Area.

Shopper account setup:

Degree the knowledge

To arrange the knowledge, entire the next steps:

Obtain the zipped archive report to make use of for this answer and unzip the recordsdata in the community.

The AWS Glue task makes use of the glue-job.py script to accomplish ETL and populates the curated desk within the Knowledge Catalog.

Create an S3 bucket referred to as consumer-configbucket-<ACCOUNT_ID> by means of the Amazon S3 console within the person account, the place ACCOUNT_ID is your AWS account ID.
Add the script to this location.

Create a connection to Salesforce

Practice the relationship setup steps defined in right here. Please make a remark of the Salesforce connector identify.

Create a connection to Salesforce within the person account

Practice the relationship setup steps defined in Create Alternative Object Glide.

Arrange assets with AWS CloudFormation

We supplied two AWS CloudFormation templates to create assets: one for the manufacturer account, and one for the patron account.

Amazon S3 now applies server-side encryption with Amazon S3 controlled keys (SSE-S3) as the bottom stage of encryption for each and every bucket in Amazon S3. Beginning January 5, 2023, all new object uploads to Amazon S3 are robotically encrypted at no further charge and without a have an effect on on efficiency. We use this default encryption for each manufacturer and person S3 buckets. If you select to carry your personal keys with AWS Key Control Carrier (AWS KMS), we propose relating to Replicating items created with server-side encryption (SSE-C, SSE-S3, SSE-KMS) for cross-account replication.

Release the CloudFormation stack within the person account

Letâs get started with developing assets within the person account. There are a couple of dependencies at the person account assets from the manufacturer account. To release the CloudFormation stack within the person account, entire the next steps:

Check in to the patron accountâs AWS CloudFormation console within the goal Area.
Make a selection Release Stack.
Make a selection Subsequent.
For Stack identify, input a stack identify, akin to stack-appflow-consumer.
Input the parameters for the connector identify, object, and manufacturer (supply) account ID.
Make a selection Subsequent.
At the subsequent web page, make a choice Subsequent.
Evaluation the main points at the ultimate web page and make a choice I recognize that AWS CloudFormation may create IAM assets.
Make a selection Create stack.

Stack advent takes roughly 5 mins to finish. It’ll create the next assets. You’ll be able to to find them at the Outputs tab of the CloudFormation stack.

ConsumerS3Bucket â consumer-databucket-<person account identity>
Shopper S3 Goal Folder â marketo-leads-source
ConsumerEventBusArn â arn:aws:occasions:<area>:<person account identity>:event-bus/consumer-custom-event-bus
ConsumerEventRuleArn â arn:aws:occasions:<area>:<person account identity>:rule/consumer-custom-event-bus/consumer-custom-event-bus-rule
ConsumerStepFunction â arn:aws:states:<area>:<person account identity>:stateMachine:consumer-state-machine
ConsumerGlueCrawler â consumer-glue-crawler
ConsumerGlueJob â consumer-glue-job
ConsumerGlueDatabase â consumer-glue-database
ConsumerAppFlow â arn:aws:appflow:<area>:<person account identity>:circulate/consumer-appflow

Manufacturer account setup:

Create a connection to Marketo

Practice the relationship setup steps defined in right here. Please make a remark of the Marketo connector identify.

Release the CloudFormation stack within the manufacturer account

Now letâs create assets within the manufacturer account. Entire the next steps:

Check in to the manufacturer accountâs AWS CloudFormation console within the supply Area.
Make a selection Release Stack.
Make a selection Subsequent.
For Stack identify, input a stack identify, akin to stack-appflow-producer.
Input the next parameters and go away the remaining as default:
- AppFlowMarketoConnectorName: identify of the Marketo connector, created above
- ConsumerAccountBucket: consumer-databucket-<person account identity>
- ConsumerAccountBucketTargetFolder: marketo-leads-source
- ConsumerAccountEventBusArn: arn:aws:occasions:<area>:<person account identity>:event-bus/consumer-custom-event-bus
- DefaultEventBusArn: arn:aws:occasions:<area>:<manufacturer account identity>:event-bus/default
Make a selection Subsequent.
At the subsequent web page, make a choice Subsequent.
Evaluation the main points at the ultimate web page and make a choice I recognize that AWS CloudFormation may create IAM assets.
Make a selection Create stack.

Stack advent takes roughly 5 mins to finish. It’ll create the next assets. You’ll be able to to find them at the Outputs tab of the CloudFormation stack.

Manufacturer AppFlow â producer-flow
Manufacturer Bucket â arn:aws:s3:::producer-bucket.<area>.<manufacturer account identity>
Manufacturer Glide Final touch Rule â arn:aws:occasions:<area>:<manufacturer account identity>:rule/producer-appflow-completion-event
Manufacturer Step Serve as â arn:aws:states:<area>:<manufacturer account identity>:stateMachine:ProducerStateMachine-xxxx
Manufacturer Step Serve as Function â arn:aws:iam::<manufacturer account identity>:function/service-role/producer-stepfunction-role

After a hit advent of the assets, move to the patron account S3 bucket, consumer-databucket-<person account identity>, and replace the bucket coverage as follows:

{
    "Model": "2008-10-17",
    "Observation": [
        {
            "Sid": "AllowAppFlowDestinationActions",
            "Effect": "Allow",
            "Principal": {"Service": "appflow.amazonaws.com"},
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Useful resource": [
                "arn:aws:s3:::consumer-databucket-<consumer-account-id>",
                "arn:aws:s3:::consumer-databucket-<consumer-account-id>/*"
            ]
        }, {
            "Sid": "Manufacturer-stepfunction-role",
            "Impact": "Permit",
            "Fundamental": {
                "AWS": "arn:aws:iam::<producer-account-id>:function/service-role/producer-stepfunction-role"
            },
            "Motion": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Useful resource": [
                "arn:aws:s3:::consumer-databucket-<consumer-account-id>",
                "arn:aws:s3:::consumer-databucket-<consumer-account-id>/*"
            ]
        }
    ]
}

Validate the workflow

Letâs stroll during the circulate:

Evaluation the Marketo and Salesforce connection setup within the manufacturer and person account respectively.

Within the structure segment, we prompt scheduling the AppFlow (producer-flow) within the manufacturer account. Then again, for speedy trying out functions, we show how one can manually run the circulate on call for.

Cross to the AppFlow (producer-flow) within the manufacturer account. At the Filters tab of the circulate, make a choice Edit filters.
Make a selection the Created At date vary for which you’ve information.
Save the variability and make a choice Run circulate.
Evaluation the manufacturer S3 bucket.

AppFlow generates the recordsdata within the producer-flow prefix inside of this bucket. The recordsdata are briefly situated within the manufacturer S3 bucket below s3://<producer-bucket>.<area>.<account-id>/producer-flow.

Evaluation the EventBridge rule and Step Purposes state mechanical device within the manufacturer account.

The Amazon AppFlow task of entirety triggers an EventBridge rule (arn:aws:occasions:<area>:<manufacturer account identity>:rule/producer-appflow-completion-event, as famous within the Outputs tab of the CloudFromation stack within the Manufacturer Account), which triggers the Step Purposes state mechanical device (arn:aws:states:<area>:<manufacturer account identity>:stateMachine:ProducerStateMachine-xxxx) within the manufacturer account. The state mechanical device copies the recordsdata to the patron S3 bucket from the producer-flow prefix within the manufacturer S3 bucket. As soon as report reproduction is entire, the state mechanical device strikes the recordsdata from the producer-flow prefix to the archive prefix within the manufacturer S3 bucket. You’ll be able to to find the recordsdata in s3://<producer-bucket>.<area>.<account-id>/archive.

Evaluation the patron S3 bucket.

The Step Purposes state mechanical device within the manufacturer account copies the recordsdata to the patron S3 bucket and sends an occasion to EventBridge within the person account. The recordsdata are situated within the person S3 bucket below s3://consumer-databucket-<account-id>/marketo-leads-source/.

Evaluation the EventBridge rule (arn:aws:occasions:<area>:<person account identity>:rule/consumer-custom-event-bus/consumer-custom-event-bus-rule) within the person account, which must have induced the Step Serve as workflow (arn:aws:states:<area>:<person account identity>:stateMachine:consumer-state-machine).

The AWS Glue crawler (consumer-glue-crawler) runs to replace the metadata adopted via the AWS Glue task (consumer-glue-job), which curates the knowledge via making use of the Don’t name filter out. The curated recordsdata are positioned in s3://consumer-databucket-<account-id>/marketo-leads-curated/. After information curation, the circulate is began as a part of the state mechanical device.

Evaluation the Amazon AppFlow task (arn:aws:appflow:<area>:<person account identity>:circulate/consumer-appflow) run standing within the person account.

Upon a a hit run of the Amazon AppFlow task, the curated information recordsdata are moved to the s3://consumer-databucket-<account-id>/marketo-leads-processed/ folder and Salesforce is up to date with the leads. Moreover, the entire authentic supply recordsdata are moved from s3://consumer-databucket-<account-id>/marketo-leads-source/ to s3://consumer-databucket-<account-id>/marketo-leads-archive/.

Evaluation the up to date information in Salesforce.

You’re going to see newly created or up to date leads created via Amazon AppFlow.

Blank up

To wash up the assets created as a part of this submit, delete the next assets:

Delete the assets within the manufacturer account:
- Delete the manufacturer S3 bucket content material.
- Delete the CloudFormation stack.
Delete the assets within the person account:
- Delete the patron S3 bucket content material.
- Delete the CloudFormation stack.

Abstract

On this submit, we confirmed how you’ll be able to give a boost to a cross-account fashion to replace information between other companions with other SaaS integrations the usage of Amazon AppFlow. You’ll be able to enlarge this concept to give a boost to more than one goal accounts.

For more info, consult with Simplifying cross-account get entry to with Amazon EventBridge useful resource insurance policies. To be informed extra about Amazon AppFlow, discuss with Amazon AppFlow.

In regards to the authors

Ramakant Joshi is an AWS Answers Architect, that specialize in the analytics and serverless area. He has a background in instrument building and hybrid architectures, and is captivated with serving to shoppers modernize their cloud structure.

Debaprasun Chakraborty is an AWS Answers Architect, that specialize in the analytics area. He has round two decades of instrument building and structure enjoy. He’s captivated with serving to shoppers in cloud adoption, migration and technique.

Suraj Subramani Vineet is a Senior Cloud Architect at Amazon Internet Services and products (AWS) Skilled Services and products in Sydney, Australia. He makes a speciality of designing and construction scalable and cost-effective information platforms and AI/ML answers within the cloud. Out of doors of labor, he enjoys taking part in football on weekends.