Setting up and deploying the Replication components

The Lambda function retrieves Amazon DocumentDB credentials from AWS Secrets Manager, sets up a connection to the Amazon DocumentDB cluster, reads the change events from the Amazon DocumentDB using change streams, and replicates them to an Amazon S3 bucket. In the Amazon S3 bucket, there will be one folder for each collection. The function also stores a change stream resume token in the Amazon DocumentDB cluster so it knows where to resume on its next run. To automate the solution, we poll for changes every 120 seconds. We use Amazon EventBridge to trigger a message to Amazon SNS, which invokes the function.

The Lambda function uses three variables that you can tune:

  • Timeout – The duration after which the Lambda function times out. The default is set to 120 seconds.
  • Documents_per_run – The variable that controls how many documents to scan from the change stream with every function run. The default is set to 1000.
  • Iterations_per_sync – The variable that determines how many iterations the Lambda function waits before syncing the resume token (the resume token to track the events processed in the change stream). The default is set to 15.

Replication Setup Instructions

From your AWS Cloud9 environment, execute the following steps to setup the replication components:

  1. Create the S3 Bucket where the replication code will reside:
    export S3BucketReplicationCode=<Enter bucket name>
    aws s3 mb s3://$S3BucketReplicationCode
    
  2. Package and upload replication code to S3.
    wget https://s3.amazonaws.com/ee-assets-prod-us-east-1/modules/25b46cda932a48f4ba4c6bdb34cd949b/v1/package_lambda.sh
    sh package_lambda.sh $S3BucketReplicationCode
    
    
  3. Create the S3 Bucket where Amazon DocumentDB events will be streamed:
    export S3BucketEvents=<Enter bucket name>
    aws s3 mb s3://$S3BucketEvents
    
  4. Choose the Launch Stack button below.
Link Description
Launch CloudFormation Template Use this link to provision additional components for Change Streams
  1. Ensure the correct AWS region is selected in the top right corner.
  2. Fill the remaining parameters for the Stack. In your Cloud9 environment, you find a jsonData.json file with most of the values you need; find the value for each parameter there using the same name in the stack. There are 2 additional parameters related to the buckets you created earlier.
  3. Check the ‘I acknowledge that AWS CloudFormation might create IAM resources’ and Choose on Create stack.

AWS Cloud Formation Template