Profiler

Amazon DocumentDB has a profiler that logs any database operations that take longer than some period of time (e.g. 100 ms). You can use the profiler logs to identify queries that are taking a long time to run.

Enabling the profiler

In order to enable the profiler, you need to use a custom Amazon DocumentDB cluster parameter group. You can follow this guide to create a new parameter group, or follow these steps:

  • In the navigation pane on the Amazon DocumentDB console, choose Parameter groups.
  • In the Cluster parameter groups pane, choose Create.
  • In the Create cluster parameter group pane, enter the following:
    • Group name - Provide a name for the new group.
    • Description - Provide a description for the new group.
  • Click Create.

Go to the Parameter groups section of the Amazon DocumentDB console and select the parameter group you just created.

Select the profiler setting, then click Edit.

Cluster Parameter Edit

Change the value to enabled.

Cluster Parameter Edit Confirm

If you like, you can also change the value of profiler_threshold_ms, which defaults to 100 ms.

Also enable audit_logs, which will be used in the next section.

Cluster Parameter Audit

Now go to the Clusters part of the console and select the checkbox by your cluster. Click Actions -> Modify.

Cluster

In the Cluster options section, select your new parameter group.

Cluster Options

In the Log exports section, enable Audit logs and Profiler logs.

Cluster Options

On the next screen, choose Apply immediately.

Cluster Options

It’ll take a few minutes for the new cluster parameter group assignment to go into effect. Once your cluster shows as available again, you need to reboot the instances. You can do this by going to the Clusters section of the Amazon DocumentDB console, selecting an instance, and choosing Reboot from the Actions menu.

Cluster Reboot

Putting load on the database

In order to generate operations that take more than 100 ms, you will use a sample data set from the Amazon DocumentDB samples repository. This data set contains example public health records about disease occurrences.

Navigate to the AWS Cloud9 management Console and choose open IDE to launch the AWS Cloud9 environment. Download the data set once you have a command prompt open in Cloud9:

wget https://raw.githubusercontent.com/aws-samples/amazon-documentdb-samples/master/datasets/cases.json

Now import the data. In these commands, note that DatabaseUser and DatabasePassword are the values you specified for your Amazon DocumentDB database when you created it. DbEndpoint is available on the console, as described in the Query Cluster section.

mongoimport --db case --collection cases -h $docdbEndpoint -u $docdbUser -p $docdbPass --ssl --sslCAFile rds-combined-ca-bundle.pem --file cases.json

Now will query the recently imported data using Mongo shell. Run the following command in Cloud9 Environment to start the mongo shell:

mongo --ssl --host $docdbEndpoint:27017 --sslCAFile rds-combined-ca-bundle.pem --username $docdbUser --password $docdbPass

After that run the following command:

use case
db.cases.find()
db.cases.find({Country_Region:"US"}).count()
db.cases.aggregate([  {$match:{Province_State:"Washington",   
  Admin2:"King", Case_Type:"Deaths"}},  {$group: {_id:null, sum: {$sum: 
  "$Difference"}}} ])

Viewing profiler logs

After you enable the profiler, you can see the logs by going to the Amazon CloudWatch console, selecting Logs -> Log groups in the navigation pane, and searching for the Log group /aws/docdb/<DbClusterIdentifier>/profiler.

Amazon CloudWatch Logs

Click on the log group and you will see a log stream for the affected database instance(s).

Amazon CloudWatch Log Stream

Click on the log stream and you will see the log contents.

Amazon CloudWatch Log Event

Log queries

You can do simple log filtering by typing a search into the Log events search box. For example, to restrict the log events to a single collection:

Amazon CloudWatch Log Filter

For more advanced analysis, you can use Amazon CloudWatch Logs Insights and run some of the sample queries in the Amazon DocumentDB Developer Guide.

For example, let’s go to the Insights part of the Amazon CloudWatch console.

Amazon CloudWatch Log Insights

Enter /aws/docdb/ into the search box to start filtering the available log groups, and select the log group for your database.

Amazon CloudWatch Log Insights Select

Enter this query:

filter ns="case.cases" | sort millis desc | limit 10

Click Run query.

Amazon CloudWatch Log Insights Results

You can also try this query:

stats count(*) by execStats.stage

This query shows you long-running queries grouped by stage. Switch to the Visualization tab and set the chart type to Bar.

Amazon CloudWatch Log Insights Chart