If you are running applications with unpredictable workloads which require frequent open and close database connections and you also want to maintain high availability during transient database failure, the Amazon RDS proxy is the solution!
It minimizes application disruption from outages affecting the availability of DB, by automatically connecting to a new DB instance while preserving application connections.
This reduces failover times for Aurora and RDS databases by up to 66%. This can help with database efficiency as well as application scalability. In this blog, we will explore Amazon RDS proxy, how it works, its use cases, etc.
We will also see how to create RDS Proxy, how to connect RDS instances using RDS Proxy, and how to monitor RDS Proxy Connections.
In this blog, we will cover:
- Challenges in connecting to RDS
- Amazon RDS Proxy solves these challenges
- Prerequisites to implement Amazon RDS Proxy
- What is Amazon RDS Proxy
- How does it work?
- Benefits of RDS proxy
- Use Cases
- Supported Databases
- AWS Services Used
- What is AWS Secrets Manager?
- How Does the AWS Secrets Manager Work?
- Create AWS Secrets for Database credentials
- Hands-on
- Conclusion
Challenges in connecting to RDS
- For each database connection, the DB server consumes memory and compute resources.
- AWS Lambda-based serverless applications make thousands of database connections in a short period of time
- Database server client consumes a lot of resources for database connection and cannot allocate enough resources for query execution
Amazon RDS Proxy solves these challenges
As it is a fully managed and highly available database proxy for Amazon RDS, Amazon RDS Proxy can solve database connection challenges.
- AWS Lambda-based modern applications can make thousands of database connections in seconds.
- Database resources can be used efficiently by using the warm connection pool of RDS Proxy. It scales automatically according to the load
- With the conventional connection method, the resources of the database server are easily exhausted.
- It is compatible with various database engines like Aurora MySQL, RDS MySQL, Aurora DB Cluster, and PostgreSQL.
- DB credentials are managed by AWS Secrets Manager and are not hard-coded into the application for improved security
Prerequisites to implement Amazon RDS Proxy
We should have the following set of networking resources before implementing the Amazon RDS Proxy:
- An Amazon Virtual Private Cloud (Amazon VPC)
- Two or more subnets across different Availability Zones,
- An Amazon RDS database and Amazon EC2 instances within the same VPC,
- An Internet gateway.
What is Amazon RDS Proxy?
Amazon RDS proxy is a fully managed, highly available database proxy for RDS Instances that makes applications more secure, scalable, and resilient to database failures. Many applications can create more connections to the databases which will result in CPU and memory exhaustion.
With the help of RDS proxies, We can allow applications to pool and share the established database connections which will result in database efficiency and application Scalability.
Amazon RDS Proxy – How does it work?
Benefits of RDS Proxy
- The RDS proxy reduces the time it takes for Aurora and RDS DB instances to failover by 66%.
- AWS Secrets Manager and IAM Role integration can be used to control database access.
- Improved application scale by pooling and sharing database connections
- Increase the application’s accessibility.
Use Cases of RDS Proxy
- For applications hosted on AWS Lambda, RDS proxy can be used (Serverless).
- For apps hosted on EC2 Instances, RDS proxy can be used.
In modern applications, connecting from AWS Lambda to a relational database is the most powerful use case for RDS Proxy.
Supported Databases
The following databases are supported by Amazon RDS proxy:
- Aurora MySQL.
- RDS MySQL 5.6 and 5.7
- Aurora DB Cluster
- PostgreSQL DB instance
Connection Management
RDS Proxy allows you to reuse a connection after each transaction in your session by default. Multiplexing is the term for transaction-level reuse.
Borrowing a connection is when RDS Proxy momentarily pulls a connection from the connection pool in order to reuse it. RDS Proxy restores that connection to the connection pool when it is safe to do so.
RDS Proxy can’t always be sure that reusing a database connection outside of the current session is safe. It keeps the session on the same connection until the session expires in these instances. Pinning is the term for this fallback behavior.
AWS Services Used
- AWS Secrets Manager
- IAM (Identify and Access Management)
- RDS (Relational Database Services)
What is AWS Secrets Manager?
AWS Secrets Manager helps you protect secrets required to access your applications, services, and IT resources.
The service enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.
How does AWS Secrets Manager work?
- Use Secrets Manager to store, rotate, monitor, and control access to secrets such as database credentials, API keys, and OAuth tokens.
- Enable secret rotation using built-in integration for MySQL, PostgreSQL, and Amazon Aurora on Amazon RDS. You can also enable rotation for arbitrary secrets using AWS Lambda functions.
- To retrieve secrets, you simply replace hard coded secrets in applications with a call to Secrets Manager APIs, eliminating the need to expose plaintext secrets.
Create Database Credentials in AWS Secrets Manager
AWS Secrets Manager helps to protect access to applications and services, And also we can easily rotate , manage and retrieve DB credentials, API keys, and other secrets.
To store the database credentials, we must first create AWS secrets.
We should use the same login and password that we used while creating an RDS Instance.
We create secrets using the username and password fields in Secrets Manager. The RDS proxy will be able to connect to the appropriate use of the Database instance as a result of this.
Hands-On
To create Secrets for the Database connection, Login to AWS Secrets Manager Console.
And then click Store a new secret
Here We are going to specify the Type of Secret we are going to use. We are going to store secrets for RDS, Choose Credentials for the RDS database
And then provide the actual RDS database username and password.
We are using the default AWS KMS key to encrypt the secrets stored in the AWS Secrets Manager.
And then choose the Database Instance from the list. Secrets Manager retrieves the connection strings about the Databases by querying the Chosen database.
and then click Next
Provide a Name for the Secret and add a short description.
Optionally, add tags for the Secrets and then click Next
We can optionally configure automatic rotation of Secrets.
We can set a rotation interval for each secret we create.
and then click Next, Under the review page, Based on the secrets configuration, We will be provided a code for various runtimes on how we can use those secrets in your applications.
And Click Store.
We have successfully added a database and its credentials in Secrets.
Creating IAM Policy:
Next, we need to create an IAM role with necessary permission for the RDS Proxy to access Database credentials from AWS Secrets Manager.
For this, we need an ARN of the secret which we created earlier.
Go to AWS Secrets manager console, select the Secret, There you can find the Secret ARN, Make a note of it.
To create an IAM role, Login to the IAM console, From the left navigation pane, choose Policies.
and click Create policy.
Click JSON, replace the existing content with the below contents.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"secretsmanager:GetRandomPassword",
"secretsmanager:CreateSecret",
"secretsmanager:ListSecrets"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "secretsmanager:*",
"Resource": [
"your_secret_ARN"
]
}
]
}
your_secret_ARN = replace with the actual Secrets ARN.
and click Review policy, and provide a name for the policy and then click Create policy.
Creating IAM Role:
Now we have to create a Role and then attach the policy with it.
To create a Role, In the Navigation pane, Choose Roles.
Click Create Role, Choose RDS as AWS Service and then click RDS – Add Role to Database and click Next:Permissions
and then search for the policy we have created and select it ,
provide a name for the role
and click Create role.
Creating RDS Proxy:
We are ready with the Secrets and the required permissions for the RDS Proxy setup.
To create an RDS proxy, Login to the RDS Console.
In the left navigation pane, Choose Proxies
And click Create proxy
Provide a name for the proxy and then choose the DB engine you’re running.
If you’re using MySQL RDS Instance choose MYSQL, If PostgreSQL RDS Instance, Choose POSTGRESQL.
Enable Require Transport Layer Security, If you want the proxy to enforce SSL/TLS connections for all the client connections.
Idle client connection timeout, The time period that the client connection can be idle before the proxy closes the connection.
The default connection timeout is 30 minutes.
For Target group configuration,
Choose RDS instances or Aurora clusters to access through this proxy.
So for this, We must create one RDS instance or Aurora cluster which has a compatible DB engine, Engine version, and other settings. For example, I have created an Aurora-MySQL Cluster.
Connection pool maximum connections, The percentage of the max_connections value the RDS proxy can use for its connections.
If you are using only a proxy instance they set the percentage to 100.
Include reader endpoint, If your proxy is associated with a single-writer Aurora cluster, you can enable a second endpoint that’s read-only. This endpoint points to the reader instances in your cluster. Using this endpoint for your query-intensive applications helps you to take advantage of the read scalability of an Aurora cluster through your proxy connections.
Under Connectivity, Select the Secrets that we have created using AWS Secrets Manager.
For IAM Role, select the IAM role which was created earlier.
For IAM Authentication, keep the default setting of Disabled.
Then for Subnets, Minimum of 2 subnets are required from different Availability zones.
By Enhanced logging for the RDS Proxy, we can gather the details of queries processed by the proxy and which are logged and published to Cloudwatch Logs.
and then click Create proxy
Once the proxy is created, we can find RDS Proxy details by clicking the proxy name where we can find the RDS Proxy endpoint.
You will find the read-only endpoint if you have enabled it, including the reader endpoint during RDS Proxy creation.
Make a note of it, We are going to use it in the next step.
Connecting to Databases Using RDS Proxy:
You connect to an RDS DB instance or Aurora DB cluster through a RDS Proxy in generally the same way as you connect directly to the database.
The main difference is that you specify the RDS proxy endpoint instead of the instance or cluster endpoint.
Once the proxy endpoint is ready, Check whether the endpoint is reachable from the EC2 Instance where the applications are hosted.
nc -zv rds-proxy-endoint 3306
It will respond as shown below.
telnet rds-proxy-endoint 3306
Connecting to an RDS instance using RDS proxy:
mysql -h rds-proxy-endpoint -u username -p
We have successfully established a connection with the RDS instance or the Cluster which is running behind the Amazon RDS proxy.
Monitoring RDS Proxy Connections:
With the help of Amazon Cloudwatch Metrics, We can monitor the Client and Database connections of the RDS Proxy.
Login to Amazon Cloudwatch Console, In the Navigation pane, choose Metrics.
Choose RDS, Per-proxy metrics.
Search for the RDS proxy name and then choose Database Connections and Client connections.
For the Display option, Choose Number.
The ClientConnections metric shows the current number of client connections to the RDS Proxy reported every minute.
The DatabaseConnections metric shows the current number of database connections from the RDS Proxy reported every minute.
Conclusion
We have successfully implemented an Amazon RDS proxy to manage database connections to improve the availability of the application. With the RDS Proxy, we can improve database efficiency and application scalability and the failover times are reduced by 66%. We will discuss more security, scalability, and database efficiency in our upcoming blogs. Stay tuned to keep getting all updates about our upcoming new blogs on AWS and relevant technologies.
Meanwhile …
Keep Exploring -> Keep Learning -> Keep Mastering
This blog is part of our effort towards building a knowledgeable and kick-ass tech community. At Workfall, we strive to provide the best tech and pay opportunities to AWS-certified talents. If you’re looking to work with global clients, build kick-ass products while making big bucks doing so, give it a shot at workfall.com/partner today.
