What should be simple quickly turns into friction, which raises Cloud/DevOps's Teams tasks with
• Choosing the right infrastructure between ECS, App Runner, or EC2
• Setting up IAM roles and networking
• Estimating costs in spreadsheets and rightsizing as per requirements
• Figuring out and writing CloudFormation, Terraform, or CDK modules before a feature reaches users.

In February 2026, AWS introduced Agent Plugins for AWS: a new way to let AI coding assistants like Claude Code and Cursor take over the heavy lifting of cloud deployment.

Instead of stitching everything together manually, you can now simply say:

“Deploy this app to AWS.”

From there, the Agent Plugin will,
• Understands your codebase
• Recommends the right AWS services
• Estimates cost before deployment
• Generates infrastructure-as-code
• And executes the deployment

All through a single, natural language interaction.

The Bigger Shift

• What we’re witnessing isn’t just a new tool, it’s a fundamental shift.

• The long-standing gap between “writing code” and “running code in the cloud” is collapsing in real time.

• Agent Plugins don’t just reduce friction, they redefine the workflow.

• And for DevOps Team and Cloud architects, this changes everything.

What exactly are AWS agent plugins?

Think of agent plugins as skill packs for your AI coding assistant. Your AI agent is smart, but it doesn't natively know the best AWS service for your specific app, the current price of App Runner per vCPU-hour, or the CDK pattern for a containerized Node backend with RDS.

An agent plugin bundles all that domain expertise into a reusable, versioned package. If your AI agent is a brilliant but generalist contractor, agent plugins are the specialist subcontractors they can now call on. The AWS plugin is your cloud architect on speed dial, available instantly, always up to date, and it never charges by the hour.

When you install the deploy-on-aws plugin, your coding agent gains:
Agent Skills: Structured deployment workflows and best practices
MCP Servers: Live AWS documentation, pricing, and IaC guidance
Hooks: Guardrails and automated validations
References: Configuration defaults and curated knowledge

Evolution of Deployments before and after Agent Plugin

Beyond individual productivity, this matters at the team level. Instead of every engineer independently pasting AWS guidance into prompts (with wildly different results), teams can standardize on shared plugins that encode approved patterns, security posture, and cost guardrails. Deterministic and dependency at scale, that's what architects really want.

Architecture breakdown: how it actually works

Here's the end-to-end flow

The three MCP servers powering the initial release do different jobs:

Hands-on walkthrough: from zero to deployed

Let's get practical. Here's how to go from nothing to a deployed Express.js + React + PostgreSQL app on AWS using the deploy-on-aws plugin.

Step 1 Prerequisites
Before installing, confirm you have:

Verify AWS CLI is configured $ aws sts get-caller-identity

Confirm Claude Code or Cursor is installed
$ claude --version

Step 2: Install the plugin

In Claude Code:

\( /plugin marketplace add awslabs/agent-plugins \) /plugin install deploy-on-aws@awslabs-agent-plugins

In Cursor:
Open Settings → Plugins → search "aws" → click Add to Cursor.

There are other AWS Plugins available in Cursor to explore and use

Pro tip: Follow the principle of least privilege when configuring your AWS credentials. Create a scoped IAM role for the agent rather than using your root or admin credentials.

Step 3: The magic prompt

Open your project in Claude Code or Cursor and type:

Deploy this Express app to AWS

Step 4: Watch the 5-step flow execute

1 Analyze: Scans codebase: identifies Express.js, Node 20.x, PostgreSQL dependency, React build, env vars, expected traffic.

2 Recommend: Selects: App Runner (backend), RDS PostgreSQL (database), CloudFront + S3 (React frontend), Secrets Manager (credentials). With rationale for each choice.

3 Estimate: Shows live monthly cost breakdown using real AWS pricing. You see the number before anything is provisioned.

4 Generate: Produces CDK TypeScript, Dockerfile, migration scripts, env config, and a GitHub Actions CI/CD pipeline.

5 Deploy: On your confirmation, provisions all resources, deploys the container, sets up CloudFront, stores secrets, and hands you URLs + CloudWatch dashboard links.

Real-world use cases beyond "deploying an app"

Architect-level perspective: where does this fit?

Comparing Agent Plugins against the existing IaC ecosystem is a fair question. Here's how I think about it:

AWS Deployment Methodology Comparison

Agent Plugins are not a Terraform replacement. They're best understood as a fast entry point. Use them to go from idea to running infrastructure quickly, then promote the generated CDK code into your standard IaC pipeline for ongoing management. Hybrid workflows will dominate.

Think of agent plugin output as a starting point, not a final commit. Run the generated CDK through your security scanning tools (Checkov, cfn-nag) and have a senior engineer review before merging to production. The plugin accelerates; human judgment validates.

Limitations and Pitfalls to watch out for

Important: AI-generated infrastructure outputs should always be reviewed before deployment against your security requirements, cost constraints, and resilience targets. The plugin is an accelerator, not an authority.

1. Blind trust in AI recommendations. The agent picks a reasonable default architecture, but it doesn't know your organization's compliance requirements, existing VPC setup, or Reserved Instance commitments. Always contextualize.

2. Skipping the cost estimate review. The plugin shows you estimated costs. Please actually read them. A misconfigured RDS Multi-AZ instance for a side project can be a nasty surprise.

3. No cost monitoring after deployment. Set up AWS Budgets alerts on day one. AI-deployed infrastructure is real infrastructure; billing doesn't care how it got there.

4. Deploying with over-privileged IAM credentials. Use a scoped IAM role with only the permissions the deployment actually needs. Least privilege always.

To Summarise

AWS is shipping more agent plugins in the coming weeks the deploy-on-aws is just the start. Think about what a full ecosystem looks like: plugins for cost optimization, security auditing, incident response, database tuning, and multi-region failover.

In two to three years, I expect "AI as DevOps co-pilot" to be the norm in Infrastructure engineering, not the exception. The interesting question isn't whether AI will write more infrastructure code, it's what the new high-value human skills will become.

Agent Plugins for AWS don't replace cloud architects, they give every developer on your team a cloud architect's first draft. What you do with that draft is still up to you.

AWS Storage Gateway bridges your on-premises infrastructure with Amazon's cloud storage — letting you use familiar protocols like NFS, SMB, and iSCSI while your data lives safely in S3, EBS, or Glacier.

What Is AWS Storage Gateway?

AWS Storage Gateway is a hybrid storage solution that acts as a bridge that lets you use Amazon S3, Glacier, and EBS storage from your local servers and applications without completely overhauling your existing setup. Your On-Prem servers speak traditional storage protocols, and the gateway quietly converts those requests into AWS API calls for accessing stored data in S3, Glacier, or EBS. It connects your on-site physical or virtual machines to nearly unlimited cloud storage in AWS. It supports popular storage classes, including S3 Standard, S3 Infrequent Access, S3 Glacier, and Glacier Deep Archive. Because it must communicate with AWS in real time, a stable internet connection is required.

In today’s hybrid IT environments, many organizations need a smooth way to connect their on-premises infrastructure with scalable cloud storage. AWS Storage Gateway serves exactly this purpose.

Types of Storage Gateway

Three distinct gateway types, each designed for a different storage use case:

File Gateway

The File Gateway exposes S3 buckets as NFS (v3 / 4.1) or SMB (v2 / v3) shares. Your operating system sees a normal network drive; under the hood, every file becomes an S3 object.

Multi-site sharing: Deploy one File Gateway VM in Data Center 1 and another in Data Center 2, both pointing at the same S3 bucket. A file uploaded from DC1 becomes visible in DC2 after the RefreshCache API call kicks off a re-inventory on the second gateway.

File Gateway This is the most commonly used option for file-based workloads. It allows you to mount Amazon S3 buckets as standard network file shares using NFS (versions 3 and 4.1) or SMB (versions 2 and 3) protocols. When files are written to the share, they are stored as individual objects in S3. Features like object versioning automatically create new versions when files are modified, deleted, or renamed. This gateway is ideal for file servers, application data, and any scenario where you want to treat cloud storage like a local network drive.

Volume Gateway

The Volume Gateway works with iSCSI the protocol that makes a remote disk feel local. Your servers mount volumes just like physical hard drives, but storage lives in AWS. If your applications require block-level storage, Volume Gateway is the right choice. It presents storage volumes to your servers over the iSCSI protocol.

Stored Volumes: Your local disk is the primary storage (low latency reads/writes), and data is asynchronously backed up to S3 as EBS snapshots. All primary data stays on your local disks, while AWS takes asynchronous point-in-time snapshots to S3 (as EBS snapshots). This provides low-latency local access with cloud backup. Volumes range from 1 TB to 16 TB.

Cached Volumes: S3 is the primary storage. Only frequently-accessed data is cached locally on fast EBS volumes, while the full dataset resides in S3. This is the more cost-effective option since S3 pricing is lower than EBS. Maximum size per volume: 32 TB. This approach helps reduce on-premises storage costs significantly.

Volume Gateway is popular for databases, ERP systems, and other applications that traditionally use SAN storage.

Tape Gateway

For enterprises running tape-based backup workflows, Tape Gateway is a drop-in replacement. It presents a virtual tape library (VTL) over iSCSI — your existing backup software (Veeam, Veritas, etc.) keeps working unchanged, but instead of physical tapes going into a shelf, data flows directly into S3 Glacier or S3 Glacier Deep Archive.

This approach removes the cost and operational burden of physical tape hardware while preserving your existing backup schedule, retention policies, and tooling.

Tape Gateway Designed for organizations with existing tape-based backup processes. Tape Gateway emulates a virtual tape library. It connects via iSCSI and stores virtual tapes in S3 Glacier or Glacier Deep Archive for long-term, low-cost retention. This option allows companies to retire physical tape hardware while keeping their current backup software and workflows almost unchanged.

How AWS Storage Gateway Works ?

The core of the solution is a lightweight Storage Gateway Appliance. A virtual machine you deploy in your environment.

Here’s the basic data flow:

• You install and activate the gateway appliance (either as a VM or hardware appliance).

• The appliance creates a local cache (minimum recommended size is around 150 GB) to store recently used or frequently accessed data for fast performance. When applications write data, it first goes to the local cache. The gateway then asynchronously uploads the data to the appropriate AWS service (S3 for files/objects, EBS snapshots for volumes, or Glacier for tapes).

• For File Gateway, you can use the RefreshCache operation to ensure all gateways see the latest files when working in multi-gateway setups. This caching mechanism delivers low latency for active data while leveraging the durability and scalability of AWS cloud storage.

Pricing

AWS Storage Gateway follows the standard AWS pay-as-you-go model. Costs depend on:

• Gateway usage — charged per gateway per month

• S3 storage — depends on storage class chosen and number of requests, billed per GB/month

• EBS snapshots — charged per GB for any snapshots taken from Volume Gateway

• AWS Region — pricing varies by geographic region

The most cost-effective approach for infrequently accessed data is pairing File Gateway with S3 Infrequent Access or Glacier storage classes. Always check the official AWS pricing page for current rates.

Deployment Options: Supported Host Platforms

Storage Gateway is delivered as a virtual appliance (OVA image) that you deploy on your existing hypervisor. No new hardware needed in most cases.

1. Virtual Appliance: Download the OVA template and run it on Supported platforms: VMware ESXi
   Microsoft Hyper-V (2012 R2+)
   Linux KVM
   Amazon EC2

2. Hardware Appliance: AWS offers a pre-configured physical device for environments where virtualization isn’t preferred.

Recommended resources for the VM include at least 16 GB RAM and 4 vCPUs, plus dedicated disks for the cache and upload buffer.

Advantages of AWS Storage Gateway

• Zero hardware changes. Your existing servers, applications, and storage protocols continue working exactly as before. Support for existing protocols (NFS, SMB, iSCSI)

• True hybrid storage. On-premises and cloud storage connects seamlessly. no forklift upgrades or migration required.

• Smooth cloud migration path. Start with a gateway, gradually shift workloads to AWS at your own pace. Simplified data migration to AWS.

• Built-in data protection through snapshots, versioning, and disaster recovery.

• Cost optimization by moving cold data to cheaper storage classes

How to Deploy AWS Storage Gateway

The walkthrough below covers setting up a File Gateway on VMware ESXi and connecting an Ubuntu Linux machine to an S3 bucket over NFS. You'll need an AWS account and an ESXi host.

Step 1 — Download the VM Image

In the AWS Console, navigate to Services → Storage Gateway → Create Gateway.

1

2

The Create Gateway wizard opens. On step 1, choose File Gateway.

1

On the next step, select VMware ESXi as the host platform, then click Download image. The file will be named something like aws-storage-gateway-latest.ova. Keep the browser tab open — you will return to finish activation later.

1

Step 2 — Deploy the Virtual Appliance on ESXi

In VMware vSphere Client, right-click your ESXi host and choose Actions → Deploy OVF Template.

Minimum requirements for File Gateway: 16 GB RAM · 4 vCPUs · one 80 GB disk (OS) · one additional 150 GB disk (cache)

1. Select OVF template. Choose "Local file" and browse to your downloaded .ova file.

   

2. Name and folder. Give the VM a descriptive name (e.g. aws-storage-gateway) and pick a vCenter inventory folder.

   

3. Compute resource. Select the ESXi host with enough free CPU and RAM.

   

4. Review details. Verify the template configuration before proceeding.

   

5. Select storage. Pick a datastore with sufficient space. Use Thick Provisioned format for the virtual disk for best performance.

   

6. Select networks. Attach the VM to a vSwitch connected to the internet.

   

7. Ready to complete. Review everything and hit Finish.
   Wait until the Storage Gateway VM is deployed from the template. You can see the job status in the Recent Tasks toolbar in vSphere Client.

   

8. Once the VM is deployed, you can see the VM name you have defined before in the list of VMs of the appropriate ESXi host (10.10.10.90 in our case).
   Right click the VM (aws-storage-gateway is the name of the Storage Gateway VM deployed from the template in this example) and in the context menu hit Edit Settings.

   

9. Add a new virtual hard disk for cache(150 GB cache disk).
   This virtual disk is used to store recently accessed files and files that are accessed frequently to reduce latency when accessing that data.
   After deployment, right-click the VM → Edit Settings → Add New Device → Hard Disk. Set the size to 150 GB with Thick Provisioning.

   

   Make sure that time is set correctly on the Storage Gateway VM, ESXi hosts, and vCenter servers. Time on the VM must be synchronized to avoid issues and for successful gateway activation.
   click Edit Settings > VM Options > VMware Tools > Synchronize guest time with host” checkbox. Hit OK to save settings.

   

10. Testing network connectivity

It is recommended to test the network connection of the Amazon Storage Gateway running as a VM locally with AWS cloud storage.
Power on the Storage Gateway VM > Log into the AWS Appliance VM by using the default credentials.

Enter the IP address of the VM (the Storage Gateway virtual appliance), not the external (WAN) IP of your router.

Click Connect to gateway.

1. Activate gateway. Activation of the gateway securely associates your gateway with your AWS account.
   Select the gateway time zone. Enter the gateway name, for example Storage Gateway AWS.
   - The name can be different from the name of the VM and the DNS name of the VM (appliance).
   - Remember that TCP 80 port must be opened on the gateway VM.

Click Activate gateway and wait until the cache disks are identified.

Configure local disks. Ensure that your 150-GB virtual disk is allocated to cache. Then hit Configure logging.

Now the File Gateway has been successfully created and it is running.

Creating a file share

It’s time to create a file share in order to connect to a bucket by using standard NFS or SMB (CIFS) protocols. Let’s configure the connection to an Amazon S3 bucket via NFS.

Squash level. Click Edit in the Mount options and select All squash to make sure that everything will work properly.

The NFS file share is created on your file gateway.

Conclusion

AWS Storage Gateway removes the friction between traditional on-premises infrastructure and the AWS cloud. Rather than rebuilding your storage architecture from scratch, you can continue using the protocols your systems already rely on — NFS and SMB for file-level access to S3, iSCSI for block-level access to EBS volumes, and virtual tape libraries as a seamless replacement for physical tape hardware.

This guide walked through the core concepts behind each gateway type and demonstrated a complete File Gateway deployment on VMware ESXi, culminating in a live NFS connection from Ubuntu Linux to an Amazon S3 bucket.

Beyond manual file transfers, Storage Gateway fits naturally into any backup strategy. Because it exposes standard NFS, SMB, and iSCSI interfaces, virtually any backup tool can use it as a target — giving organizations a straightforward path to offloading backup data to AWS without changing their existing workflows.

Overview of Azure Identities and Governance

Microsoft Entra ID Users and Groups

Microsoft Entra ID (formerly Azure AD) provides cloud-based identity services for managing users and groups. Key tasks include creating internal and external users, assigning licenses, and enabling self-service features like password reset to reduce admin overhead.

Access Management with RBAC

Azure Role-Based Access Control (RBAC) allows granular permissions using built-in or custom roles assigned at various scopes, helping interpret and audit access for security.

Subscriptions and Governance Strategies

Tools like Azure Policy, resource locks, tags, and management groups enable the enforcement of standards, cost monitoring, and hierarchical organization across subscriptions

AZ-104: A Comprehensive Guide to Managing Azure Identities and Governance (20-25%)

This detailed guide explores the core components of the "Manage Azure identities and governance" domain in the AZ-104 Microsoft Azure Administrator certification.
I will be documenting my learnings and explaining my understanding of each topic, based on hands-on lab experience gained from below,
https://microsoftlearning.github.io/AZ-104-MicrosoftAzureAdministrator/

This Blog is for Lab 01
https://microsoftlearning.github.io/AZ-104-MicrosoftAzureAdministrator/Instructions/Labs/LAB_01-Manage_Entra_ID_Identities.html

Manage Microsoft Entra users and groups

Microsoft Entra ID is a cloud-based identity and access management service essential for Azure security. Unlike on-premises Active Directory, it supports tenants (isolated directories), subscriptions (billing boundaries), and users/groups for access control. Administrators can create tenants, add users and groups, and manage roles to control resources on Azure.

Create users and groups
Creating users and groups is essential for managing access. Users can be internal (member users) or external (guests), and groups consist of security groups for permissions and Microsoft 365 groups for collaboration.

Steps to create a user in the Azure portal:
- Navigate to Microsoft Entra ID > Users > New user.
- Enter user principal name (e.g., user@abcd.com), display name, and password.
- Add Identity properties > Assign roles or groups > then create.
- For bulk creation, go to Entra ID > Users > Bulk Create. Use CSV templates or PowerShell.

Steps to create a group:
- Go to Microsoft Entra ID > Groups > New group.
- Choose type (Security or Microsoft 365), name, description, and membership
type (Assigned, Dynamic User/Device).
- Add owners/members, then create.

Best practices: Use dynamic groups for automatic membership based on attributes (e.g., department=Sales); limit direct user assignments to favor group-based access for scalability. Example: Create a "FinanceTeam" security group and add users for RBAC assignments.

Manage user and group properties
Properties include display name, job title, department, and usage location (for licensing). Update via portal, PowerShell, or Azure API.
- Select a user/group in Entra ID.
- Edit properties section; for groups, manage membership or ownership.

Best practices: Regularly audit properties for compliance; use bulk updates for large-scale changes.

Manage licenses in Microsoft Entra ID
Licenses enable features like Microsoft 365 or Azure AD Premium. Assign via direct or group-based methods.
- Go to Entra ID > Licenses > All products.
- Select a product (e.g., Microsoft 365 E5), then assign to users/groups.
- For group-based: Create a group, assign licenses to it; members get inherited.

Best practices: Monitor usage reports to avoid over-licensing; use dynamic groups for auto-assignment based on roles.

Example: Assign Azure AD P1 licenses to a "SSPR-Enabled" group for self-service features.

Manage external users
External users (B2B guests) collaborate via invitations or self-service sign-up. Configure settings to control access levels, invitations, and domain restrictions.
Steps to invite a guest:
- In Entra ID > Users > New guest user
- Enter email, send invitation

Configuration in external collaboration settings:
- Guest user access: Limited (default) to restrict directory enumeration.
- Invite permissions: Limit to admins or specific roles like Guest Inviter.
- Collaboration restrictions: Allow/deny specific domains.

Best practices: Enable self-service sign-up for apps; use cross-tenant access settings for B2B; audit sign-in logs for security.

Example: Restrict invitations to @partner.com domain for controlled collaboration.

Configure self-service password reset (SSPR)
SSPR allows users to reset passwords without admin help, requiring P1/P2 licenses.
Step-by-step guide:

1. In Entra admin center > Entra ID > Password reset > Properties.

2. Enable for Selected group (e.g., SSPR-Test-Group).

3. Set authentication methods (e.g., email, mobile app code; require 2 methods).

4. Require registration on sign-in; set reconfirmation every 180 days.

5. Enable notifications for password resets and admin alerts.

6. Customize helpdesk link (e.g., support@contoso.com).

Best practices: Start with pilot group; combine with MFA for security; monitor usage via reports.
Example: Use incognito browser to register methods at aka.ms/ssprsetup, then reset at aka.ms/sspr.

Manage access to Azure resources

Azure RBAC provides fine-grained access using roles, scopes, and assignments. It integrates with Entra ID principals (users, groups, managed identities).

Manage built-in Azure roles
Built-in roles like Owner (full access), Contributor (create/manage without access grants), and Reader (view-only). Over 100 roles available; Use custom roles for tailored permissions.

Best practices: Use least privilege; prefer built-in over custom roles unless needed.

Example: Virtual Machine Contributor manages VMs but not networks/storage.

Assign roles at different scopes
Scopes: Management group, subscription, resource group, resource. Assignments inherit downward.

Steps in portal:
- Navigate to resource/scope > Access control (IAM) > Add role assignment.
- Select role (e.g., Contributor), members (user/group), and assign.

Via PowerShell:
New-AzRoleAssignment -RoleDefinitionName "Contributor" -PrincipalId <ID> -Scope <scope>.

Best practices: Assign at higher scopes for efficiency; use groups for multiple users.

Example: Assign Reader at subscription for the auditing team.

Interpret access assignments
Assignments include principal, role, scope. Effective permissions are additive;
use "Check access" in IAM to view access assignments.

Steps:
- In IAM > Role assignments, filter by principal/scope.
- Use "Deny assignments" tab for explicit denies.

Best practices: Regularly review with Azure AD access reviews; audit logs for changes.

Manage Azure subscriptions and governance

This section covers tools for organizing, securing, and optimizing subscriptions.

Implement and manage Azure Policy
Azure Policy enforces rules via definitions and initiatives. Effects: Audit, Deny, Modify, etc.

Steps to create:

1. In portal > Policy > Definitions > New definition (JSON with rules/parameters).

2. Group into initiative > Assignments > Assign to scope (e.g., subscription).

3. Remediate non-compliant resources.

Best practices: Start with Audit; use initiatives for grouped policies; manage as code.

Example: Deny non-approved VM SKUs Policy.

Configure resource locks
Locks prevent Azure Resource delete/modify.
Locks Types: CanNotDelete, ReadOnly.

Steps:
In resource > Locks > Add (name, type, notes).

Best practices: Apply at group level; consider data plane impacts.

Example: Apply CanNotDelete Lock on critical Azure Resources, and Ready-Only Locks to prevent modification.

Apply and manage tags on resources
Tags in Azure are name–value pairs applied to resources, resource groups, or subscriptions to logically organize assets and support cost management, governance, and operations.
- Do not affect resource functionality
- Tags are inherited from resource group → resource (but not vice versa)
- Tags integrate directly with Azure Cost Management: Filter costs by tag

Steps: Azure resource > Tags > Edit

Best practices: Enforce Tags via Azure Policy; use Tag based filter for billing reports.
Example: Key : Value Environment : Production).

Manage resource groups(RG)
Resource groups in Azure are logical containers that hold related resources together for management, governance, and lifecycle control.

Steps to Create and Manage a Resource Group
- Portal: Azure Portal → Resource groups → Create
- Specify Name, Region, and optional Tags
- Delete RG: Deletion requires explicit confirmation. No partial delete RG

Best Practices:
- Group resources by lifecycle (app, environment, project)
- Apply tags at the resource group level
- Use Azure Policy for governance
- Avoid mixing unrelated workloads

Manage subscriptions
Subscriptions in Azure define billing, access control, and resource limits.
Each subscription is linked to a billing account. Resources cannot span multiple subscriptions.
- Subscriptions can be transferred between billing accounts or directories
- Ownership changes should be planned to avoid access loss
- Subscriptions control billing and access, not regions
- Moving resources between subscriptions may cause downtime
- Policies and RBAC often apply at the subscription level

Steps: Portal > Subscriptions > Add (offer type, billing).

Best practices:
- Use multiple subscriptions for isolation (Prod vs Non-Prod). Separate by environment, workload, or compliance.
- Apply budgets, policies, and role assignments at subscription scope

Manage costs by using alerts, budgets, and Azure Advisor recommendations
Cost management in Azure helps monitor spending, prevent overruns, and optimize resource usage using Cost Management, budgets, and Advisor recommendations.
- Monitor Cost via Azure Cost Management + Billing
- View spend by subscription, resource group, timeframe, service, or tag
- Analyze trends and forecast usage, and set a budget for alerting.

Steps:
- Create budget: Scope > Budgets > Add (amount, alerts)
Budgets trigger alerts only—they do not stop resources.
- Alerts: Action groups for notifications
- Advisor: Azure Advisor Review recommendations
(e.g., right-size VMs, use reserved instances, remove unused resources)

Best Practices:
- Set multiple budget thresholds (e.g., 50%, 75%, 90%)
- Use tags for accurate cost allocation
- Combine budgets and alerts with Azure Policy for governance
- Review Advisor recommendations regularly

Configure management groups
Management groups in Azure provide a hierarchical structure to organize subscriptions and apply governance at scale.

- Subscriptions can be moved between management groups
- Policies and RBAC assignments inherit downward
- The root management group sits at the top of the hierarchy
- Requires User Access Administrator or Owner at root to manage hierarchy.

Steps:
- Portal: Azure Portal → Management groups → Create
- Specify Management Group ID and Display Name
- Add subscriptions or child management groups

Best practices:
- Align hierarchy with organizational structure
- Separate Prod / Non-Prod at higher levels
- Apply Azure Policy at management group scope
- Limit access at the root; delegate lower levels

Governance Tools Overview

Conclusion

Managing Azure identities and governance is foundational for secure, compliant, and cost-effective cloud operations. Mastery of Microsoft Entra ID (users, groups, external identities, licensing, and self-service features), principled access control with RBAC, and disciplined subscription governance using Azure Policy, resource locks, tags, and management groups will help you enforce standards, reduce risk, and simplify administration. This domain represents a significant portion of the AZ-104 exam (about 20–25%), so practical familiarity matters as much as theory.

Key takeaways and next steps:

• Adopt least-privilege access: use built-in roles where appropriate and create custom roles only when necessary.

• Harden identities: require MFA, enable self-service password reset, and consider Privileged Identity Management for just-in-time elevation.

• Design RBAC around scopes and role assignments to simplify auditing and reduce blast radius.

• Enforce standards with Azure Policy and use management groups to scale governance across subscriptions.

• Implement tagging and cost controls to improve visibility and chargeback.

• Practice with hands-on labs and real-world scenarios (for example, the AZ-104 lab materials) to solidify skills and prepare for the exam.

Keep practicing these concepts in real or lab environments—consistent, hands-on experience is the most effective way to become a confident Azure administrator.

Key references read,

• https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/az-104

• https://learn.microsoft.com/en-us/training/modules/manage-users-and-groups-in-aad

• https://microsoftlearning.github.io/AZ-104-MicrosoftAzureAdministrator/Instructions/Labs/LAB_01-Manage_Entra_ID_Identities.html

• https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/az-104

• https://learn.microsoft.com/en-us/entra/identity/authentication/tutorial-enable-sspr

• https://learn.microsoft.com/en-us/azure/role-based-access-control/overview

• https://learn.microsoft.com/en-us/entra/external-id/external-collaboration-settings-configure

• https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal

• https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/lock-resources

• https://learn.microsoft.com/en-us/azure/governance/policy/overview

• https://learn.microsoft.com/en-us/azure/governance/management-groups/overview

Thank you for the read. Hope you like it. I appreciate your feedback.

Follow for more Azure and AWS Content. Happy Learning!

Regards,
Jineshkumar Patel

What Are Target Tracking Scaling Policies?

Target tracking scaling policies allow you to define a target value for a specific Amazon CloudWatch metric (e.g., CPU utilization, request count per instance). EC2 Auto Scaling then adds or removes instances to keep the metric close to your target. For example:

• If your target is 60% CPU utilization, Auto Scaling will launch instances when CPU usage exceeds 60% and terminate them when it drops below.

• If your application handles HTTP requests, you could target 500 requests per minute per instance to ensure consistent throughput.

This "set-it-and-forget-it" approach reduces manual intervention and ensures your application scales seamlessly.

Benefits of Target Tracking Scaling

1. Automatic Scaling: Responds to traffic spikes or drops without manual input.

2. Cost Optimization: Runs only the instances needed, avoiding over-provisioning.

3. Simplified Configuration: No need to define complex scaling rules for every scenario.

4. Integration with CloudWatch: Leverage built-in metrics or custom metrics for granular control.

5. Proactive Scaling: Adjusts capacity before performance degrades.

How Target Tracking Scaling Works

1. Select a Metric: Choose a CloudWatch metric (e.g., CPUUtilization, RequestCountPerTarget).

2. Set a Target Value: Define the ideal average value for the metric.

3. Auto Scaling Adjusts Capacity: EC2 Auto Scaling uses Amazon’s scaling algorithms to add/remove instances.

While in the creation of Auto-Scaling-Group > Configure Group Size and Scaling Policies (Screenshot Above)

Step-by-Step Implementation

Example 1: Scaling Based on CPU Utilization

Scenario: You want to maintain an average CPU utilization of 40% for your web servers.

Via AWS Console:

1. Navigate to EC2 Auto Scaling Groups > Select your ASG > Automatic Scaling tab.

2. Click Create dynamic scaling policy > Target tracking scaling.

3. Configure:

   • Policy name: cpu80-target

   • Metric type: Average CPU Utilization

   • Target value: 80

4. Click Create.

Screenshot of the metric selection and target value fields.

AWS Documentation Suggested Steps as follows,

To create a target tracking scaling policy for an existing Auto Scaling group

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/, and choose Auto Scaling Groups from the navigation pane.

2. Select the check box next to your Auto Scaling group.

   A split pane opens up in the bottom of the page.

3. Verify that the scaling limits are appropriately set. For example, if your group's desired capacity is already at its maximum, you need to specify a new maximum in order to scale out. For more information, see Set scaling limits for your Auto Scaling group.

4. On the Automatic scaling tab, in Dynamic scaling policies, choose Create dynamic scaling policy.

5. To define a policy, do the following:

   1. For Policy type, keep the default of Target tracking scaling.

   2. Specify a name for the policy.

   3. For Metric type, choose a metric. You can choose only one metric type. To use more than one metric, create multiple policies.

      If you chose Application Load Balancer request count per target, choose a target group in Target group.

   4. Specify a Target value for the metric.

   5. (Optional) For Instance warmup, update the instance warmup value as needed.

   6. (Optional) Select Disable scale in to create only a scale-out policy. This allows you to create a separate scale-in policy of a different type if wanted.

6. Choose Create.

AWS CLI Command to set target-tracking-configuration based of “Average CPU Utilization”

aws autoscaling put-scaling-policy \  
  --policy-name cpu80-target \  
  --auto-scaling-group-name my-asg \  
  --policy-type TargetTrackingScaling \  
  --target-tracking-configuration '{  
    "PredefinedMetricSpecification": {  
      "PredefinedMetricType": "ASGAverageCPUUtilization"  
    },  
    "TargetValue": 80.0  
  }'

I had a Stress test running on my EC2 instances, which gets my EC2 instances CPU utilization over 80%

which can be seen under EC2 > AutoScalingGroup Settings > MyAutoScalingGroup > Monitoring (Screenshot Below)

Click on Activity to see New EC2 Instance getting created due to “Target Tracking CPU Utilization Policy”

Example 2: Scaling Based on Application Load Balancer (ALB) Requests

Scenario: Your application uses an ALB, and you want to maintain 1000 requests per minute per instance.

Via AWS Console:

1. Follow steps 1–2 above.

2. Configure:

   • Metric type: Application Load Balancer Request Count Per Target

   • Target value: 1000

   • ALB Target Group: Select your target group.

AWS CLI command for “Application Load Balancer Request Count Per Target”

aws autoscaling put-scaling-policy \  
  --policy-name alb-requests-target \  
  --auto-scaling-group-name my-asg \  
  --policy-type TargetTrackingScaling \  
  --target-tracking-configuration '{  
    "PredefinedMetricSpecification": {  
      "PredefinedMetricType": "ALBRequestCountPerTarget",  
      "ResourceLabel": "app/my-alb/1234567890abcdef/targetgroup/my-target-group/abcdef0123456789"  
    },  
    "TargetValue": 1000.0  
  }'

Advanced Tips

1. Custom Metrics: Use CloudWatch custom metrics (e.g., queue depth, memory usage) for niche workloads.

2. Cooldown Periods: Configure cooldown timers to prevent rapid scaling fluctuations.

3. Combination with Other Policies: Pair target tracking with step scaling for complex scenarios.

Troubleshooting issues to consider

• Metric Not Found: Ensure the metric is correctly associated with your ASG (e.g., ALB metrics require the ASG to be attached to the target group).

• Over-scaling: Adjust the target value or enable instance warm-up to stabilize scaling decisions.

• Monitor CloudWatch Alarms: Target tracking creates CloudWatch alarms automatically—check their status in the AWS Console.

Conclusion

Target tracking scaling policies are a game-changer for managing EC2 workloads. By automating capacity adjustments, you save time, reduce costs, and ensure reliable performance. Whether you’re scaling based on CPU, network, or custom metrics, this feature simplifies infrastructure management.

Reference: Explore the official AWS documentation for advanced configurations.

You are now ready to optimize your EC2 fleet !
Implement target tracking, and let AWS handle the heavy lifting!

Thank you for the read. Hope you like it. I appreciate your feedback.

Follow for more Azure and AWS Content. Happy Learning!

Regards,
Jineshkumar Patel

Example Illustration of EBS Multi-Attach (Image Reference - AWS)

When to Use EBS Multi-Attach

Scenarios & Use Cases

1. Clustered Databases: Deploy databases like SQL Server Failover Clusters or Oracle RAC where multiple nodes require access to shared data.

2. High Availability Applications: Applications needing redundant compute nodes with shared storage for failover.

3. Real-Time Collaboration: Workloads requiring simultaneous read/write access to shared data (with proper file system coordination).

Prerequisites

• Volume Type: Only Provisioned IOPS SSD (io1/io2) volumes support Multi-Attach.

• Availability Zone: All attached instances must reside in the same AZ.

• File System: Use a cluster-aware file system (e.g., GFS2, Windows Failover Cluster) to avoid data corruption.

• Instance Compatibility: Instances must support EBS encryption if the volume is encrypted.

Step-by-Step Guide to Configuring EBS Multi-Attach

1. Creating a Multi-Attach Volume

Using the AWS Console

1. Navigate to the EC2 Console → Volumes → Create Volume.

2. Configure:

   • Volume Type: io1 or io2

   • Size & IOPS: Define size (GiB) and provisioned IOPS.

   • Availability Zone: Match the AZ of your target instances.

   • Multi-Attach: Check Enable Multi-Attach.

   • (Optional) Configure encryption or tags.

3. Click Create Volume.

Using AWS CLI

aws ec2 create-volume \
  --volume-type io2 \
  --size 100 \
  --iops 3000 \
  --availability-zone us-east-1a \
  --multi-attach-enabled

2. Attaching the Volume to Multiple Instances

After creation, attach the volume to multiple instances in the same AZ.

Using AWS CLI

# Attach to Instance 1 Replace with your instance ID
aws ec2 attach-volume \
  --volume-id vol-12345abc \
  --instance-id i-0abcd1234 \
  --device /dev/sdf

# Attach to Instance 2 Replace with your instance ID
aws ec2 attach-volume \
  --volume-id vol-12345abc \
  --instance-id i-0efgh5678 \
  --device /dev/sdf

3. Enabling Multi-Attach on Existing io2 Volumes

Note: Only io2 volumes can have Multi-Attach enabled post-creation (if unattached).

Console Method

1. Select the volume → Actions → Modify Volume.

2. Check Enable Multi-Attach → Modify.

AWS CLI Command

aws ec2 modify-volume \
  --volume-id vol-12345abc \
  --multi-attach-enabled

Example Scenario: SQL Server Failover Cluster

Imagine deploying a SQL Server Failover Cluster across two EC2 instances (i-0abcd1234 and i-0efgh5678) in us-east-1a.

1. Create a Multi-Attach Volume:

    aws ec2 create-volume --volume-type io2 --size 500 --iops 5000 \
      --availability-zone us-east-1a --multi-attach-enabled
   

2. Attach to Both Instances:

    # Attach to primary instance
    aws ec2 attach-volume --volume-id vol-12345abc --instance-id i-0abcd1234 --device /dev/sdf
   
    # Attach to secondary instance
    aws ec2 attach-volume --volume-id vol-12345abc --instance-id i-0efgh5678 --device /dev/sdf
   

3. Configure Windows Failover Cluster:

   • Initialize the disk as NTFS or ReFS on the primary instance.

   • Add the disk to the cluster shared volumes (CSV) in Windows Server Failover Cluster Manager.

Disabling Multi-Attach

To disable Multi-Attach, ensure the volume is attached to no more than one instance.

Console Method

1. Select the volume → Actions → Modify Volume.

2. Uncheck Enable Multi-Attach → Modify.

AWS CLI Command

aws ec2 modify-volume \
  --volume-id vol-12345abc \
  --no-multi-attach-enabled

Best Practices & Limitations

Best Practices

• Cluster-Aware File Systems: Always use file systems designed for concurrent access (e.g., GFS2, CSV).

• I/O Coordination: Applications must handle concurrent writes to avoid data corruption.

• Monitoring: Use Amazon CloudWatch to track volume performance (VolumeQueueLength, BurstBalance).

Limitations

• Volume Types: Only io1 and io2 support Multi-Attach.

• AZ Bound: All instances must be in the same AZ.

• io1 Restriction: Multi-Attach cannot be enabled on io1 after creation.

• Encryption: Encrypted volumes require compatible instances.

• The following table shows volume modification support for Multi-Attach enabled io1 and io2 volumes after creation.

* You can't enable or disable Multi-Attach while the volume is attached to an instance.

Troubleshooting

• Attachment Failure: Verify AZ alignment, instance support for encryption, and volume type.

• Data Corruption: Ensure the application or file system handles concurrent writes.

Conclusion

EBS Multi-Attach is a game-changer for clustered workloads needing shared storage. By following this guide, you can configure Multi-Attach for io1/io2 volumes, attach them to multiple instances, and deploy highly available applications. Remember to use cluster-aware file systems and monitor performance to avoid bottlenecks.

Further Reading:

• AWS EBS Documentation

• Clustered File Systems for Linux - Attach an EBS volume to multiple EC2 instances using Multi-Attach

By mastering EBS Multi-Attach, you unlock new possibilities for building resilient, high-performance clustered File System architectures on AWS.

I hope this blog post has been helpful. If you have any further questions or encounter any issues, please feel free to leave a comment below.

Thank you for reading! Happy Learning!

Like and Follow for more content.

Thank you,
Jineshkumar Patel

In today’s digital age, storing files in the cloud is a must. AWS S3 offers a secure and scalable storage solution, and combining it with AWS SDK for Python (Boto3) in Flask and a bit of front-end magic gives you a lightweight S3 file uploader that can be integrated into any web project. Whether you’re a developer looking to expand your toolkit or a tech enthusiast wanting a hands-on project, this guide is for you.

What You’ll Build

We will create a web application that:

• Allows users to select and upload files.

• Uses JavaScript (with progress events) to display upload progress.

• Uses AWS SDK for Python (Boto3) in Flask as the backend to communicate with AWS S3.

• Displays a list of uploaded files retrieved from S3.

• Shows the bucket name to ensure that the correct S3 bucket is being used.

Files in the Project:

• app.py – The main Python Flask application.

• index.html – The front-end HTML interface.

• script.js – JavaScript for handling uploads and dynamic updates.

• style.css – CSS for styling the app.

• .env – Environment variables for AWS credentials and bucket name.

Key components:

1. Client-facing UI with drag-and-drop capabilities to upload Files

2. Real-time file list synchronization. List uploaded Files to S3 Bucket

3. S3File Upload Progress Bar

4. Error Handling

Settings up file/folder structure in VS Code

Pre-requisites:

1. AWS Account and Access to AWS Management Console.

2. S3 Bucket - Create a unique named S3 Bucket from AWS Console.

3. Create Access Key for newly Create “s3-uploader-user” IAM User (Screenshot Below)

4. “s3-uploader-user” permission access policy AmazonS3FullAccess

5. Installed Python , VSCode on your local machine.

Pre-requisites 2 , 3 Screenshot references

Note: AmazonS3FullAccess is not necessary. You can only give required granular permissions only as well.

Setting Up the Backend (app.py)

Below is the code for our Python app that’s uses AWS SDK for Python (Boto3).

This script creates several routes:

• / to serve the main page.

• /upload (and /api/upload) to handle file uploads.

• /list to retrieve the list of files stored in S3.

• /bucket to return the bucket name for verification.

import os
from flask import Flask, render_template, request, jsonify
import boto3
from botocore.exceptions import ClientError

app = Flask(__name__)

# Load AWS credentials and bucket name from environment variables.
AWS_ACCESS_KEY = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
BUCKET_NAME = os.environ.get('AWS_S3_BUCKET') or 'your-bucket-name'

# Create an S3 client using the boto3 library.
s3_client = boto3.client(
    's3',
    aws_access_key_id=AWS_ACCESS_KEY,
    aws_secret_access_key=AWS_SECRET_KEY
)

# Home route renders the index.html template.
@app.route("/")
def index():
    return render_template("index.html")

# File upload route.
@app.route("/upload", methods=["POST"])
def upload():
    if "file" not in request.files:
        return jsonify({"error": "No file part in the request"}), 400

    file = request.files["file"]
    if file.filename == "":
        return jsonify({"error": "No selected file"}), 400

    try:
        # Upload the file to the specified S3 bucket.
        s3_client.upload_fileobj(file, BUCKET_NAME, file.filename)
        return jsonify({"message": "File uploaded successfully", "file": file.filename})
    except ClientError as e:
        return jsonify({"error": str(e)}), 500

# List files stored in the S3 bucket.
@app.route("/list", methods=["GET"])
def list_files():
    try:
        response = s3_client.list_objects_v2(Bucket=BUCKET_NAME)
        files = [obj["Key"] for obj in response.get('Contents', [])]
        return jsonify({"files": files})
    except ClientError as e:
        return jsonify({"error": str(e)}), 500

# Return the bucket name.
@app.route('/bucket', methods=['GET'])
def get_bucket_name():
    return jsonify({"bucketName": BUCKET_NAME})

# A second API endpoint for uploading (if needed).
@app.route('/api/upload', methods=['POST'])
def api_upload():
    if "file" not in request.files:
        return jsonify({"error": "No file part in the request"}), 400

    file = request.files["file"]
    if file.filename == "":
        return jsonify({"error": "No selected file"}), 400

    try:
        s3_client.upload_fileobj(file, BUCKET_NAME, file.filename)
        return jsonify({"message": "File uploaded successfully", "file": file.filename})
    except ClientError as e:
        return jsonify({"error": str(e)}), 500

if __name__ == "__main__":
    app.run(debug=True)

Key Points

• AWS Credentials: The app uses environment variables (.env file) to securely store AWS keys and the bucket name.

• S3 Client: The boto3 client is initialized with your credentials, allowing file uploads to S3.

• Routes:

  • The /upload endpoint checks for a file in the request and uploads it using upload_fileobj.

  • The /list endpoint retrieves all objects (files) in the bucket.

  • The /bucket endpoint returns the bucket name, which is useful for debugging and ensuring your app is connected to the correct S3 bucket.

Front-End Implementation: index.html & script.js

The front end of our app provides a user-friendly interface for uploading files and viewing the list of uploaded files. Although the full index.html isn’t listed in the original snippet, here’s an example based on the provided JavaScript code.

index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>AWS S3 File Upload</title>
    <link rel="stylesheet" href="style.css">
</head>
<body>
    <header>
        <img src="logo.png" alt="App Logo">
        <!-- Your App / Brand Logo for this Web App Reference as .png file with its location -->
        <h1>Upload Your Files to AWS S3</h1>
    </header>

    <div id="bucketDisplay">Bucket Name: Loading...</div>

    <!-- File Upload Form -->
    <form id="uploadForm">
        <input type="file" id="fileInput" name="file">
        <button type="submit">Upload File</button>
    </form>

    <!-- Progress Bar -->
    <div id="progressContainer">
        <div id="progressBar"></div>
    </div>

    <!-- Files List Table -->
    <table id="filesTable">
        <thead>
            <tr>
                <th>Uploaded Files</th>
            </tr>
        </thead>
        <tbody id="filesBody">
            <!-- JavaScript will dynamically insert file names here -->
        </tbody>
    </table>

    <script src="script.js"></script>
</body>
</html>

Code Explanation for index.html

• Header: Displays a logo and title. Replace logo.png with your actual logo.

• Bucket Display: A div with the id bucketDisplay will show the name of your S3 bucket (retrieved via the /bucket API).

• File Upload Form:

  • The form with id="uploadForm" contains an <input> for file selection.

  • A submit button to trigger the upload.

• Progress Bar:

  • Wrapped inside a container (#progressContainer) with an inner div (#progressBar) that visually represents the upload progress.

• Files Table:

  • A table with an id filesTable is set up to display the list of uploaded files. The body (#filesBody) is populated dynamically via JavaScript.

• Script Inclusion: The script.js file is included at the end of the body to handle the dynamic functionality.

Diving Into script.js

The JavaScript file is responsible for:

• Handling the file upload via AJAX.

• Displaying the upload progress.

• Dynamically listing the files stored in the S3 bucket.

• Fetching and displaying the S3 bucket name.

script.js

const uploadForm = document.getElementById('uploadForm');
const fileInput = document.getElementById('fileInput');
const progressBar = document.getElementById('progressBar');
const filesBody = document.getElementById('filesBody'); // Table body for files

// Function to list files from the S3 bucket.
function listFiles() {
    fetch('/list')
        .then(response => response.json())
        .then(data => {
            filesBody.innerHTML = ''; // Clear any previous file entries
            if (data.files) {
                data.files.forEach(file => {
                    const row = document.createElement('tr');
                    const cell = document.createElement('td');
                    cell.textContent = file;
                    row.appendChild(cell);
                    filesBody.appendChild(row);
                });
            }
        });
}

// Function to fetch and display the bucket name.
function fetchBucketName() {
    fetch('/bucket')
        .then(response => response.json())
        .then(data => {
            document.getElementById('bucketDisplay').textContent =
                "Bucket Name: " + data.bucketName;
        })
        .catch(console.error);
}

// Event listener for the upload form.
uploadForm.addEventListener('submit', function (e) {
    e.preventDefault();
    const file = fileInput.files[0];
    if (!file) return;

    const formData = new FormData();
    formData.append("file", file);

    const xhr = new XMLHttpRequest();
    xhr.open("POST", "/upload", true);

    // Update the progress bar during the file upload.
    xhr.upload.addEventListener("progress", function (evt) {
        if (evt.lengthComputable) {
            const percentComplete = (evt.loaded / evt.total) * 100;
            progressBar.style.width = percentComplete + '%';
        }
    }, false);

    // Once upload is complete, reset the progress bar and update the file list.
    xhr.onload = function () {
        progressBar.style.width = '0%';
        if (xhr.status === 200) {
            listFiles(); // Refresh file list upon successful upload.
        } else {
            alert("Error uploading file: " + xhr.responseText);
        }
    };

    xhr.send(formData);
});

// On window load, initialize the file list and bucket name display.
window.onload = function() {
    listFiles();
    fetchBucketName();
};

Key Points

• File Upload Handling:

  • The script prevents the default form submission behavior.

  • It creates a FormData object to package the file data.

  • An XMLHttpRequest is used to post the file to the /upload endpoint.

• Progress Bar:

  • The xhr.upload.addEventListener("progress", ...) event calculates the percentage of the upload and adjusts the width of the progress bar accordingly.

• Dynamic Content:

  • listFiles() fetches the list of files from the S3 bucket and dynamically updates the table in the HTML.

  • fetchBucketName() retrieves the bucket name from the server to display it for user confirmation.

• User Experience:

  • The upload progress, immediate file list update after a successful upload, and bucket name display all contribute to a seamless user experience.

Environment Setup (.env)

Make sure to create a .env file (or set environment variables in your deployment environment) with the following content:

AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_S3_BUCKET=file-upload-s3-jpatel-aws-bucket

Replace the placeholders with your actual AWS credentials and bucket name.

Running the App

1. Install Dependencies:
   Make sure you have Flask and boto3 installed:

    pip install flask boto3 python-dotenv
   

   

2. Set Up AWS Credentials:
   Ensure your AWS credentials and bucket name are properly configured in your .env file or environment.

3. Run the S3Upload App locally:
   Start the application by running:

    python app.py
   

   Your application should now be accessible at Localhost aka http://127.0.0.1:5000/
   Note: if above two python commands runs successfully, your application will be shown on the above localhost address. Meaning, your application is working.

   

Wooaallaaa !!! Open your App successfully running on http://127.0.0.1:5000/

Conclusion

We’ve just built a fully functional file uploader that stores files directly to AWS S3 Bucket using simple AWS SDK for Python(boto3) and JavaScript! This guide covered:

• Configuring your Web App with AWS S3 using AWS SDK for Python boto3.

• Creating endpoints for file upload and file listing.

• Building a responsive front end with HTML, CSS, and JavaScript.

• Implementing real-time progress updates during file uploads.

This project not only demonstrates the power of combining these technologies but also gives you a solid foundation for integrating cloud storage into your own projects.

For more information and reference on this, please follow
Boto3 Documentation : Uploading Files

Now that we've built a robust AWS S3 file uploader with Flask and JavaScript, our journey has just begun. I encourage you to take this guide further by adapting and expanding upon it to fit your unique needs. Experiment with various hosting providers—whether it's Heroku, AWS Elastic Beanstalk, DigitalOcean, or any another host platform—to deploy your application and scale it to meet real-world demands or integrate to create API Endpoint to use it in your app.

As you host and optimize your project, keep iterating: add new features, refine your code, and tailor the experience for your users. Share your improvements and insights with the community, and collaborate with fellow developers to push the boundaries of what's possible. Your creativity and persistence are the keys to transforming this simple guide into a fully-fledged, dynamic solution.

I hope this blog post has been helpful. If you have any further questions or encounter any issues, please feel free to leave a comment below.

Thank you for reading! Happy Learning!

Like and Follow for more content.

Thank you,
Jineshkumar Patel

What is MFA Delete?

MFA Delete is a security feature that requires additional authentication for two specific operations:

1. Changing the versioning state of your S3 bucket

2. Permanently deleting an object version

When enabled, MFA Delete requires two forms of authentication:

• Your standard AWS security credentials

• A valid serial number and six-digit code from an approved MFA device

This two-factor approach significantly reduces the risk of accidental deletions or unauthorized changes to your S3 bucket's versioning state.

Why Use MFA Delete?

1. Enhanced Security: By requiring an additional authentication factor, MFA Delete adds an extra layer of protection to your S3 data.

2. Prevention of Accidental Deletions: It helps prevent unintended permanent deletions of object versions or changes to bucket versioning.

3. Compliance: For organizations with strict data protection requirements, MFA Delete can help meet compliance standards.

4. Root Account Protection: Only the AWS account root user can enable or disable MFA Delete, providing an additional safeguard for critical operations.

How to Enable MFA Delete

Enabling MFA Delete involves a few key steps:

1. Enable S3 Versioning: MFA Delete can only be enabled on versioned buckets.

2. Use AWS CLI or API: MFA Delete can't be enabled through the AWS Management Console; you must use the AWS CLI or API.

3. Use the Root Account: Only the AWS account root user can enable or disable MFA Delete.

Here's an example AWS CLI command to enable both S3 Versioning and MFA Delete:

aws s3api put-bucket-versioning --bucket YOUR-BUCKET-NAME --versioning-configuration Status=Enabled,MFADelete=Enabled --mfa "SERIAL-NUMBER MFA-CODE"

Replace "YOUR-BUCKET-NAME" with your actual bucket name, "SERIAL-NUMBER" with your MFA device's serial number, and "MFA-CODE" with the current six-digit code from your MFA device.

Example:

aws s3api put-bucket-versioning --bucket customer-pii-storage --versioning-configuration Status=Enabled,MFADelete=Enabled --mfa "arn:aws:iam::456789012345:mfa/root-account-mfa-device 234567"

Remember to replace the MFA device ARN and the six-digit MFA code with your actual values when running these commands

1. Using the AWS Management Console:

   • Sign in to the AWS Management Console

   • Navigate to the IAM service

   • Select "Users" from the left sidebar

   • Click on the user name

   • In the "Security credentials" tab, find the "Assigned MFA device" section

   • The ARN will be displayed there

The MFA device ARN typically follows this format: arn:aws:iam::ACCOUNT_ID:mfa/DEVICE_NAME

Important Considerations

• Once enabled, MFA Delete requires MFA authentication for disabling versioning or permanently deleting object versions.

• IAM users, even with full S3 permissions, cannot enable, disable, or modify MFA Delete settings.

• While MFA Delete prevents permanent deletions, IAM users with appropriate permissions can still create delete markers for objects.

Best Practices for Implementing MFA Delete

1. Set up a Backup MFA Device: To avoid losing access if your primary MFA device is lost, set up a backup device when first enabling MFA delete.

2. Use Hardware or Virtual MFA Devices: Employ approved authentication devices to generate the required six-digit code.

3. Encrypt Data: Always encrypt data at rest and in transit. Use client-side encryption or SSL/TLS for data transmission to and from S3.

4. Implement Least Privilege Access: Use IAM policies, bucket policies, and ACLs to ensure users have only the necessary permissions.

5. Enable Logging and Monitoring: Use AWS Config rules and tools like AWS Trusted Advisor to monitor your S3 setup and detect potential security issues.

6. Consider S3 Object Lock: For critical data, use S3 Object Lock to prevent object deletion or modification for a specified period.

7. Use Multi-Region Application Architecture: For critical data, consider using Cross-Region replication to enhance disaster recovery capabilities.

8. Regularly Audit Access: Maintain a thorough record of all identities with access to your S3 resources and review regularly.

Conclusion

MFA Delete is a powerful tool in your S3 security arsenal. By requiring additional authentication for critical operations, it provides an extra safeguard against data loss and unauthorized changes. While it may add a slight complexity to your workflow, the enhanced security it offers is well worth the effort, especially for sensitive or mission-critical data stored in S3. Remember, security in the cloud is a shared responsibility. By leveraging features like MFA Delete, you're taking an active role in protecting your data and ensuring the integrity of your S3 buckets. As cyber threats continue to evolve, implementing robust security measures like MFA Delete is not just a best practice—it's a necessity for maintaining the confidentiality, integrity, and availability of your valuable data assets.

I hope this blog post has been helpful. If you have any further questions or encounter any issues, please feel free to leave a comment below.

Thank you for reading! Happy Learning!

Like and Follow for more Azure and AWS content.

Thank you,
Jineshkumar Patel

Here are the Performance Settings Options when Selecting Amazon EFS File system’s Throughput mode. (Image below)

Elastic Throughput Mode (Default and AWS Recommended)

How It Works

• Automatically scales throughput performance up or down to meet workload needs

• No need to specify or provision throughput capacity

• Pay only for the amount of metadata and data read or written

• No accrual or consumption of burst credits

Best For

• Spiky or unpredictable workloads

• Performance requirements that are difficult to forecast

• Applications that drive throughput at an average-to-peak ratio of 5% or less

Considerations

• Available only for file systems using General Purpose performance mode

• Simplifies management by eliminating the need for manual throughput adjustments

Bursting Throughput Mode

How It Works

• Baseline Throughput: Scales with storage size (50 KiB/s per GiB of data stored)

• Burst Capability: Accumulate credits when below baseline, allowing temporary higher throughput

Best For

• Variable workloads with occasional traffic spikes

• Cost-efficient option for workloads not requiring sustained high throughput

Considerations

• Monitor burst credits via Amazon CloudWatch (BurstCreditBalance)

• Larger file systems (>1 TiB) can sustain longer bursts

Provisioned Throughput Mode

How It Works

• Manually define throughput (MiB/s) independent of storage size

• Pay for storage fees + hourly charges for provisioned throughput

Best For

• Steady, high-throughput needs

• Predictable workloads requiring consistently high performance

Considerations

• Use when baseline bursting limits are insufficient

• Adjust provisioned throughput as needs change

Key Differences

How to Choose

1. Start with Elastic Throughput if:

   • Your workload is unpredictable or spiky

   • You want simplified management and cost optimization

2. Consider Bursting Throughput if:

   • You have variable workloads with occasional spikes

   • Cost savings are a priority over guaranteed throughput

3. Switch to Provisioned Throughput if:

   • You need guaranteed throughput for latency-sensitive tasks

   • Your workload consistently requires high throughput

By aligning your EFS throughput mode with your application's needs, you can optimize both performance and costs.

Here's a summary of the key points:

1. Elastic Throughput: Recommended for most workloads, especially those with unpredictable or spiky performance requirements.

2. Bursting Throughput: Suitable for variable workloads with occasional traffic spikes, offering a balance between cost and performance.

3. Provisioned Throughput: Ideal for steady, high-throughput needs and predictable workloads requiring consistently high performance.

Final Tips

• Utilize Amazon CloudWatch metrics to monitor your file system's performance, including throughput usage and, if applicable, burst credit balance.

• During development, test different throughput modes to compare cost-effectiveness and performance for your specific workload1.

• Keep in mind that you can switch between throughput modes, but there's a 24-hour waiting period before making another change.

• For most use cases, start with Elastic Throughput as it automatically scales to meet workload needs without manual intervention.

• If you're unsure about your application's peak throughput requirements or have very spiky workloads, Elastic Throughput is often the best choice.

• Consider using Provisioned Throughput when you have a clear understanding of your workload's performance needs and require guaranteed throughput.

By carefully considering these factors and leveraging the appropriate throughput mode, you can ensure that your Amazon EFS file system delivers optimal performance while maintaining cost efficiency for your specific use case.

Adapted from the Amazon EFS Performance Guide: https://docs.aws.amazon.com/efs/latest/ug/performance.html

Hope you have enjoyed the Blog. Thank you for Reading.

Happy Learning.

Like and Follow for more Azure and AWS Content.

Thank you,
Jineshkumar Patel

• Sharing confidential documents with clients.

• Distributing time-sensitive media (e.g., concert recordings).

• Allowing users to download purchased software securely.

How to Grant Users Access to S3 Objects

By default, all objects stored in Amazon S3 are private, meaning only the owner of the bucket has access to them. However, if you need to grant users access to specific buckets or objects without making them publicly available, you can do so by assigning the appropriate permissions through an IAM policy. Alternatively, you can use presigned URLs to provide temporary access without requiring users to have AWS credentials or IAM permissions.

So Why to use Presigned URLs?

A presigned URL is a time-limited URL that grants temporary access to a specific S3 object. These URLs allow users to either read or write (update) an object, depending on the permissions you configure. The URL includes specific parameters set by your application, ensuring controlled and secure access. These parameters include:

• Bucket: The name of the bucket where the object is stored (or will be stored).

• Key: The name or path of the object.

• Expires: The duration for which the URL remains valid.

Once the expiration time passes, the URL becomes invalid, and the user can no longer access the object. Importantly, presigned URLs are securely signed by the S3 bucket owner, ensuring that only authorized users can interact with the object during the specified time frame.

Project Lab: Create and Test Presigned URLs
Objective: Build a system to share private S3 objects securely and temporarily using pre-signed URLs.

Prerequisites:

Prerequisites for Using aws s3 cp

1. AWS CLI Installed:
   The AWS Command Line Interface (CLI) must be installed on your machine.

• Installation instructions: AWS CLI Installation Guide.

• Verify installation:

    # Run to verify AWS CLI is installed 
    aws --version
  
    # Configure AWS CLI with AWS Credential and Default Region. 
    aws configure
  

• You’ll be prompted for:

  • AWS Access Key ID: Your IAM user’s access key.

  • AWS Secret Access Key: Your IAM user’s secret key.

  • Default Region Name: The AWS region (e.g., us-east-1).

  • Default Output Format: Optional (e.g. jason)

• Verify configuration:

    cat ~/.aws/credentials  
    cat ~/.aws/config
  

Step 1: Create an S3 Bucket and Upload a Test File

1. Create a private S3 bucket:

    aws s3api create-bucket --bucket your-unique-bucket-name --region us-east-1
   

2. Upload a sample file (e.g., medical-report.pdf):

    echo "Confidential Patient Data" > medical-report.pdf  
    aws s3 cp medical-report.pdf s3://your-unique-bucket-name/
   

Step 2: Generate a Presigned URL via AWS CLI

Generate a URL that expires in 5 minutes (300 seconds):

aws s3 presign s3://your-unique-bucket-name/medical-report.pdf --expires-in 300

https://your-unique-bucket-name.s3.amazonaws.com/medical-report.pdf?AWSAccessKeyId=...&Signature=...&Expires=...

Step 3: Generate a Presigned URL with Python (Boto3)

1. Install Boto3:

    pip install boto3
   

2. Python script (generate_presigned_url.py):

    import boto3  
    from datetime import timedelta  
   
    s3_client = boto3.client('s3')  
   
    url = s3_client.generate_presigned_url(  
        ClientMethod='get_object',  
        Params={  
            'Bucket': 'your-unique-bucket-name',  
            'Key': 'medical-report.pdf'  
        },  
        ExpiresIn=300  # 5 minutes  
    )  
   
    print("Presigned URL:", url)
   

3. Run the script:

    python generate_presigned_url.py
   

   When you run the script, it will generate a presigned URL that looks something like this:

   plaintext

   Copy

    Presigned URL: https://your-unique-bucket-name.s3.amazonaws.com/medical-report.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20231015%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231015T123456Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9s0t1u2v3w4x5y6z7a8b9c0d
   

   What Does This Mean?

   • URL Structure: The URL points to the medical-report.pdf file stored in the your-unique-bucket-name S3 bucket.

   • Temporary Access: The URL is valid for 5 minutes (as specified by ExpiresIn=300).

   • Security: The URL includes a signature (X-Amz-Signature) and other parameters that ensure only authorized users can access the file during the specified time frame.

How to Use the URL

1. Copy the pre-signed URL and share it with the intended user.

2. The user can open the URL in a web browser or use it in a tool like curl or wget to download the file.

3. After 5 minutes, the URL will expire, and access to the file will be revoked.

Step 4: Test URL Expiration

1. Generate a URL with a 10-second expiration:

    aws s3 presign s3://your-unique-bucket-name/medical-report.pdf --expires-in 10
   

2. Wait 15 seconds and try accessing the URL. You’ll see an HTTP 403 Forbidden error.

Step 5: Security Best Practices

1. Short Expiration Times: Use 5-15 minutes for most cases.

2. HTTPS Only: Always generate presigned URLs with HTTPS.

3. Restrict IAM Policies: Limit the IAM user/role to only s3:GetObject permissions.

    {  
        "Version": "2012-10-17",  
        "Statement": [{  
            "Effect": "Allow",  
            "Action": "s3:GetObject",  
            "Resource": "arn:aws:s3:::your-unique-bucket-name/*"  
        }]  
    }
   

Conclusion
Presigned URLs are a powerful way to securely share private S3 objects without exposing your bucket to the public. By following this guide, you’ve learned to generate URLs via CLI and SDK, test their expiration, and apply security best practices. Use this for secure document sharing, time-bound downloads, or even temporary access in serverless apps!

For more information and reference on this, please follow

• AWS S3 Presigned URL Documentation

• Download and upload objects with presigned URLs

• Generate a presigned URL in modular AWS SDK for JavaScript

I hope this Blog helps you understand Amazon S3 Presigned URLs' capabilities and gives you insights into the amazing new ways to share Temporary Files with Amazon S3 Objects.

For more information on this, please follow Working with Mountpoint for Amazon S3

Thank you for the read. Hope you like it.
I appreciate your time.

Follow for more Azure and AWS Content. Happy Learning!

Regards,
Jineshkumar Patel

Mountpoint for S3 translates local file system API Calls to S3 Object API Calls. Presents Amazon S3 Objects as files in the local file system. Amazon S3 Mountpoint seamlessly integrates with Linux-based systems, allowing file-aware applications to connect directly to Amazon S3 buckets without the need for complex configurations or additional software.

In the realm of cloud storage solutions, Amazon Simple Storage Service (Amazon S3) stands tall as a reliable and scalable option for businesses of all sizes. Its flexibility, durability, and cost-effectiveness have made it a go-to choice for storing vast amounts of data securely in the cloud. However, integrating Amazon S3 with Linux/Windows server applications has often presented challenges, requiring additional tools and configurations. Enter Mountpoint for Amazon S3, an open-source file client that simplifies this process, enabling seamless connectivity between Application Servers and Amazon S3 buckets.

Key Characteristics

• Mountpoint for Amazon S3 boasts high performance and high throughput

• Scalability: Whether you're dealing with small files or large datasets, Amazon S3 Mountpoint scales effortlessly to accommodate your storage operation needs, providing seamless access to Amazon S3 directly from your Compute instance.

• Cost-Effective: By eliminating the need for additional hardware or proprietary solutions, Amazon S3 Mountpoint helps reduce infrastructure costs associated with data storage and management.

• Reliability and Durability: Leveraging the robust infrastructure of Amazon S3, Amazon S3 Mountpoint offers high reliability and durability, with built-in redundancy and data protection mechanisms to safeguard against data loss or corruption.

• Encryption and Security: Amazon S3 Mountpoint prioritizes data security by supporting encryption in transit and at rest, ensuring that sensitive data remains protected throughout the transfer process.

• Use case: Mountpoint for Amazon S3 is ideal for workloads that read large datasets (terabytes to petabytes in size). Common use cases include large-scale machine learning (ML) training, autonomous vehicle simulation, genomics analysis, and image rendering. While these workloads read large datasets over several compute instances, they write sequentially to a file from a single node. This means they do not need shared file system features such as locking.

Mountpoint for Amazon S3 serves as a bridge between Linux-based systems and Amazon S3 buckets, facilitating direct access to stored data without the need for complex configurations or additional software layers. By mounting S3 buckets as local filesystems, Mountpoint enables Linux applications to interact with S3 objects as if they were traditional files and directories, streamlining data management and enhancing compatibility.

Note: For applications that require shared file system features such as file locking and POSIX permissions, you can use Amazon FSx for Lustre with a data repository association to your S3 bucket.

How it works

Demo

1. Download the Mountpoint for the Amazon S3 package "mount-s3.rpm" .

    wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
   

   

2. Install the package by using the following command:

    sudo yum install ./mount-s3.rpm
   

   

   

3. Check the Mount S3 Version

   

   I have uploaded some .jpg files onto my S3 Bucket name "mountpoint-bucket"

   

4. To use Mountpoint for Amazon S3, your host needs valid AWS credentials with access to the bucket or buckets that you would like to mount. For example, you can create a new AWS Identity and Access Management (IAM) user and role for this purpose. Make sure that this role has access to the bucket or buckets that you would like to mount.
   OR You can pass the IAM role to your Amazon EC2 instance with an instance profile. As shown below, I created a IAM Role that has access to S3 Bucket and have EC2 instance assume this role.

   

5. Run the mount-s3 command with s3 bucket ARN with instance mount directory

   run

   mkdir mntdir

   

   S3 BUCKET is MOUNTED using Mountpoint file client.

6. We can check the files (Objects) inside the S3 bucket directly from the mounted S3 mount point.

   

   from here, After you mount your bucket locally, you can use common Linux commands, such as cat or ls, to work with your S3 objects.

Unmount your bucket by using the umount command. This command unmounts your S3 bucket and exits Mountpoint.

Limitation of Mountpoint

\> Unlike FUSE, Mountpoint mounts cannot be added to your fstab file so do not persist after stopping your machine. They need to be re-mounted each time the machine is restarted.

\> Mountpoint for Amazon S3 is available only for Linux operating systems

\> Mountpoint cannot access files archived in S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive tiers.

Here in this demo, I have used Mountpoint to access the existing S3 bucket mounted on Linux VM which contained image files which I can perform action against and Copy or push to S3 to my local mount directory and seamlessly push new images to S3 bucket using mountpoint.

Hope this Blog helps understand Amazon S3 Mountpoint capabilities and give you insights about the amazing new ways to connect with S3 from your Linux(Application) Instance.

For more information on this, please follow Working with Mountpoint for Amazon S3

Thank you for the read. Hope you like it.
I appreciate your time.

Follow for more Azure and AWS Content. Happy Learning!

Regards,
Jineshkumar Patel

Detailed Monitoring Key Features

AWS detailed monitoring offers several key features that provide enhanced visibility and insights into the performance of your Amazon EC2 instances. Some of the notable features include:

1. Granular Metrics: Detailed monitoring provides more granular metrics compared to basic monitoring. It collects data at a higher frequency (1-minute intervals) for a wide range of system-level metrics, including CPU utilization, disk I/O, network traffic, and more.

2. Real-Time Visibility: With detailed monitoring, you get real-time visibility into the performance of your EC2 instances. The 1-minute data collection interval allows you to monitor changes and trends in system metrics with greater accuracy and responsiveness.

3. Enhanced Analysis: Detailed monitoring enables you to perform in-depth analysis of your instance performance over short time intervals. This granularity is particularly useful for identifying performance spikes, bottlenecks, and other issues that may require immediate attention.

4. Custom Alarms: You can create custom CloudWatch alarms based on detailed monitoring metrics to trigger notifications or automated actions when specific thresholds are breached. This allows you to proactively monitor and manage the health and performance of your EC2 instances.

5. Billing and Cost Optimization: By leveraging detailed monitoring metrics, you can gain better insights into resource utilization patterns and optimize your EC2 instance usage to improve cost-effectiveness. This helps in identifying underutilized instances or instances with excessive resource consumption.

6. Integration with AWS Services: Detailed monitoring metrics seamlessly integrate with other AWS services, such as Amazon CloudWatch, AWS CloudTrail, and AWS Config. This enables you to leverage the collected data for monitoring, troubleshooting, auditing, and compliance purposes across your AWS environment.

7. Historical Analysis: Detailed monitoring provides historical data for up to 15 months, allowing you to analyze long-term trends and patterns in your EC2 instance performance. This historical perspective is valuable for capacity planning, performance optimization, and trend analysis.

8. Flexible Monitoring Options: AWS offers flexible pricing options for detailed monitoring, allowing you to enable or disable it on a per-instance basis. This flexibility enables you to balance the level of monitoring granularity with cost considerations based on your specific requirements.

Using AWS Management Console

1. Sign in to AWS Management Console: Sign in to your AWS Management Console using your credentials.

2. Navigate to Amazon EC2 Console: Navigate to the Amazon EC2 console by clicking on the "Services" dropdown menu and selecting "EC2" under the "Compute" section.

3. Access Instances: In the navigation panel on the left-hand side, click on "Instances" under the "Instances" section.

4. Select Instance: Select the Amazon EC2 instance that you want to examine from the list of instances displayed.

5. Check Monitoring Configuration: On the selected Instance > Select "Actions" > Select the "Monitor and Troubleshoot" option and then > Select "Manage Detailed Monitoring".

6. Enable Detailed Monitoring Checkbox configuration value to Enable.
   Similarly, Uncheck for Disable Detailed Monitoring.

Save.

Using AWS CLI

1. To enable or disable detailed monitoring for your Amazon EC2 instances using the AWS Command Line Interface (CLI), you can use the aws ec2 monitor-instances and aws ec2 unmonitor-instances commands respectively. Below are the steps to perform both operations:

   Enable Detailed Monitoring:

   1. List Instance IDs: First, you need to list the IDs of the Amazon EC2 instances for which you want to enable detailed monitoring. You can use the describe-instances command with custom filters to list the IDs.

    bashCopy codeaws ec2 describe-instances \
      --region <your-region> \
      --filters Name=instance-state-name,Values=running \
      --query 'Reservations[*].Instances[*].InstanceId' \
      --output text

Replace <your-region> with the AWS region where your instances are located.

2. Enable Detailed Monitoring: Once you have the instance IDs, you can enable detailed monitoring for them using the monitor-instances command.

    bashCopy codeaws ec2 monitor-instances \
      --instance-ids <instance-id-1> <instance-id-2> ... \
      --region <your-region>

Replace <instance-id-1> <instance-id-2> ... with the IDs of the instances you want to enable detailed monitoring for.

Disable Detailed Monitoring:

1. List Instance IDs: Similar to enabling, you need to list the IDs of the Amazon EC2 instances for which you want to disable detailed monitoring.

2. Disable Detailed Monitoring: Use the unmonitor-instances command to disable detailed monitoring for the specified instances.

    bashCopy codeaws ec2 unmonitor-instances \
      --instance-ids <instance-id-1> <instance-id-2> ... \
      --region <your-region>

Replace <instance-id-1> <instance-id-2> ... with the IDs of the instances you want to disable detailed monitoring for.

Example:

Let's say you want to enable detailed monitoring for an instance with ID i-1234567890abcdef0 in the us-east-1 region:

    bashCopy codeaws ec2 monitor-instances \
      --instance-ids i-1234567890abcdef0 \
      --region us-east-1

To disable detailed monitoring for the same instance:

    bashCopy codeaws ec2 unmonitor-instances \
      --instance-ids i-1234567890abcdef0 \
      --region us-east-1

By following these steps and using the appropriate CLI commands, you can easily enable or disable detailed monitoring for your Amazon EC2 instances.

In conclusion, enabling detailed monitoring for your Amazon EC2 instances is essential for gaining deeper insights into their performance, ensuring optimal resource utilization, and proactively managing your AWS environment. With its granular metrics, real-time visibility, custom alarm capabilities, and seamless integration with other AWS services, detailed monitoring empowers you to effectively monitor, analyze, and optimize the performance of your EC2 instances. Whether it's for cost optimization, troubleshooting, or capacity planning, detailed monitoring provides the necessary data and tools to make informed decisions and ensure the smooth operation of your AWS infrastructure. By leveraging detailed monitoring, you can enhance operational efficiency, improve resource utilization, and maintain the overall health and performance of your EC2 instances, ultimately contributing to a more reliable and scalable AWS environment.

Thank you for the read. Hope you like it. I appreciate your time.

Follow for more Azure and AWS Content. Happy Learning!

Regards,

Jineshkumar Patel

1. Storage Class

Amazon S3 Glacier Instant Retrieval

Amazon S3 Glacier Instant Retrieval is designed for data archives that require immediate access. By leveraging this storage class, businesses can significantly reduce storage costs compared to standard S3 storage, while still ensuring fast retrieval times when needed.

Amazon FSx for NetApp OnTap Single AZ

Amazon FSx for NetApp OnTap Single AZ offers high-performance file storage optimized for enterprise workloads. With features like high availability and data deduplication, businesses can efficiently manage storage costs while meeting performance requirements.

2. Lifecycle Management

Amazon S3 Intelligent Tiering

Amazon S3 Intelligent Tiering automatically moves objects between two access tiers – frequent access and infrequent access – based on access patterns. This ensures that data is stored in the most cost-effective tier without sacrificing performance.

More on S3 Storage Classes Intelligent Tiering here .

The graphics above is from here.

Amazon EFS Intelligent Tiering

Similar to Amazon S3 Intelligent Tiering, Amazon EFS Intelligent Tiering optimizes storage costs by automatically moving files to infrequent access storage classes based on access patterns. Choose a Transition into IA option to move infrequently accessed files to the IA storage classes. From the drop-down list, you can choose lifecycle policies of 7, 14, 30, 60, or 90 days. Additionally, choose a Transition out of IA option and select On first access to move files back to EFS Standard or EFS One Zone storage classes on access. This ensures that frequently accessed data is stored in the most cost-effective manner.

The screenshot above is taken from here credit to Channy Yun.

To disable EFS Intelligent-Tiering, set both the Transition into IA and Transition out of IA options to None. This will disable lifecycle management, and your files will remain on the storage class they’re on.

Amazon EBS Snapshots Archive

Amazon EBS Snapshots Archive allows businesses to archive EBS snapshots to Amazon S3 Glacier, reducing storage costs while retaining data for long-term retention and compliance requirements.
Here's how Archive EBS Snapshot got created and put it into S3 Glacier.

And restore Snapshot from Archive

Screenshots taken from Amazon EBS Snapshots Archive by Sébastien Stormacq

3. Data Reduction

Compression and Deduplication on Amazon FSx Family Storage Data

By enabling compression and deduplication on Amazon FSx family storage data, businesses can significantly reduce storage costs by eliminating redundant data and optimizing storage efficiency. Here is more about how to enable Deduplication is here.

Additional Strategies

• Amazon S3 Storage Classes: Leveraging a combination of Amazon S3 storage classes, including Standard, Intelligent-Tiering, Glacier, and Glacier Deep Archive, based on data access patterns and performance requirements.

• Amazon S3 Lifecycle Policies: Implementing lifecycle policies to automatically transition objects to lower-cost storage classes or delete obsolete data based on predefined rules.

• Amazon EBS Volume Types: Selecting the appropriate EBS volume type (e.g., General Purpose SSD, Provisioned IOPS SSD, Cold HDD) based on performance requirements and cost considerations.

• AWS Data Transfer Optimization: Optimizing data transfer costs by using AWS Direct Connect or AWS DataSync for data transfer between AWS services and on-premises infrastructure.

AWS S3 Pricing here.
AWS S3 Storage Classes here.
AWS S3 Pricing Calculator here.

By implementing these strategies and leveraging AWS's comprehensive suite of storage optimization tools and services, businesses can effectively manage storage costs while ensuring performance, scalability, and data durability. As data continues to grow exponentially, optimizing storage costs on AWS remains a critical aspect of cloud infrastructure management.

In conclusion, optimizing storage costs on AWS requires a holistic approach that encompasses storage class selection, lifecycle management, data reduction techniques, and leveraging additional AWS services tailored to specific use cases and requirements. By following best practices and continuously monitoring and adjusting storage strategies, businesses can achieve significant cost savings while maintaining optimal performance and reliability for their storage workloads.

The optimal storage solution for a system varies based on the following:

• Type of access method (block, file, or object)

• Patterns of access (random or sequential)

• Required throughput

• Frequency of access (online, offline, archival)

• Frequency of update (WORM, dynamic)

• Availability and durability constraints

Characteristics such as shareable, file size, cache size, access patterns, latency, throughput, and persistence of data. Those characteristics can lead you toward the best storage solution, such as block storage, file storage, or object storage.

Determine storage characteristics

When you evaluate a storage solution, determine the available storage characteristics, such as the following:

• Ability to share the storage

• Ideal file size and maximum file size

• Storage cache size

• Average or expected latency

• Maximum throughput

• Maximum IOPS

• Persistence of data

Then match your requirements to the AWS service that best fits your needs.

Questions to help determine storage requirements

The following questions help you to segment data within each of your workloads and determine your storage requirements:

• How often and how quickly do you need to access your data? AWS offers storage options and pricing tiers for frequently accessed, less frequently accessed, and infrequently accessed data.

• Does your data store require high IOPS or throughput? AWS provides categories of storage that are optimized for performance and throughput. Understanding IOPS and throughput requirements will help you provision the right amount of storage and avoid overpaying.

• What storage access protocols are required? Pre-existing applications are often developed based on specific operating systems. The operating system can affect the access protocol. For example, Linux-based applications that require file system access usually require NFS. Windows-based applications require SMB as the protocol.

• How critical (durable) is your data? Critical or regulated data needs to be retained at almost any expense and tends to be stored for a long time.

• How sensitive is your data? Highly sensitive data must be protected from accidental and malicious changes, not only data loss or corruption. Durability, cost, and security are equally important to consider.

• How large is your dataset? Knowing the total size of the dataset helps in estimating storage capacity and cost.

• How transient is your data? Transient data is short-lived and typically does not require high durability. (Note: Durability refers to average annual expected data loss.) Clickstream and Twitter data are good examples of transient data.

• How much are you prepared to pay to store the data? Setting a budget for data storage will inform your decisions about storage options.

Make decisions based on access patterns and metrics

Choose storage systems based on your workload's access patterns. Configure them by determining how the workload accesses data. You can sometimes increase storage efficiency or increase a performance metric by choosing a different storage type. Configure the storage options you choose to match your data access patterns.

• Optimize your storage usage and access patterns – Choose storage systems based on your workload's access patterns and the characteristics of the available storage options. Determine the best place to store data so that you can meet your requirements while reducing overhead. Use performance optimizations and access patterns when configuring and interacting with data based on the characteristics of your storage (for example, striping volumes or partitioning data).

• Select appropriate metrics for storage options – Ensure that you select the appropriate storage metrics for the workload. Each storage option offers various metrics to track how your workload performs over time. Make sure that you are measuring against any storage metrics indicating peak performance and trends. For storage systems that are fixed sized, such as Amazon Elastic Block Store (Amazon EBS) or Amazon FSx, ensure that you are monitoring the amount of storage used against the overall storage size. Create automation when possible to increase the storage size when reaching a threshold.

• Monitor metrics – Amazon CloudWatch can collect metrics across the resources in your architecture. You can also collect and publish custom metrics to surface business metrics or derived metrics. Use CloudWatch or third-party solutions to set alarms that indicate when thresholds are breached.

For additional information, see Storage Architecture Selection in the AWS Well-Architected Framework.

Here is the Decision Tree illustrated below published by Adi Simon

AWS Storage Offerings

In conclusion, selecting the right storage solution in AWS is crucial for optimizing performance, cost, and resource utilization in your system architecture. Understanding the characteristics and requirements of your data, such as access patterns, throughput, durability, and sensitivity, is essential for making informed decisions about storage options.

By considering factors like access methods, frequency of access, and data persistence, you can match your storage needs to the appropriate AWS service, whether it's block storage, file storage, or object storage. Additionally, leveraging AWS metrics and monitoring tools like CloudWatch allows you to track performance, trends, and usage patterns, enabling you to optimize storage usage and make data-driven decisions for your workload. By following these best practices and principles, you can design a well-architected storage infrastructure that meets your business needs efficiently and effectively on AWS.

Thank you for the read. Hope you like it. I appreciate your time.

Follow for more Azure and AWS Content. Happy Learning!

Regards,

Jineshkumar Patel

Introduction to AWS Storage Gateway

AWS Storage Gateway is a hybrid cloud storage service that enables businesses to securely integrate their on-premises applications with AWS cloud storage services. By providing a range of gateway options, AWS Storage Gateway accommodates various use cases and data storage requirements, facilitating smooth data transfer between on-premises environments and AWS.

The Four Types of Gateways

1. Amazon S3 File Gateway: This gateway offers a seamless connection to the cloud, allowing organizations to store application data files and backup images as durable objects in Amazon S3. With support for SMB or NFS-based access and local caching, Amazon S3 File Gateway ensures efficient data transfer and accessibility.

   

2. Amazon FSx File Gateway: Designed for customers with unstructured or file data, Amazon FSx File Gateway optimizes on-premises access to fully managed, highly reliable file shares in Amazon FSx for Windows File Server. This gateway caters to low-latency requirements, providing seamless integration with SMB-based group shares and business applications.

   

3. Volume Gateway: Volume Gateway presents cloud-backed iSCSI block storage volumes to on-premises applications, offering scalable and cost-effective storage solutions. With operating modes such as cache mode or stored mode, Volume Gateway efficiently manages on-premises data in Amazon S3, ensuring data availability and durability.

   

4. Tape Gateway: Tape Gateway simplifies data backup workflows by replacing physical tapes on premises with virtual tapes in AWS. Supporting leading backup applications and caching virtual tapes on-premises, Tape Gateway ensures low-latency data access while seamlessly integrating with existing backup processes.

   

   Use Tape Gateway to replace physical tapes on premises with virtual tapes on AWS—reducing your data storage costs without changing your tape-based backup workflows. Tape Gateway supports all leading backup applications and caches virtual tapes on premises for low-latency data access. It compresses your tape data, encrypts it, and stores it in a virtual tape library in Amazon Simple Storage Service (Amazon S3). From there, you can transfer it to either Amazon S3 Glacier Flexible Retrieval or Amazon S3 Glacier Deep Archive to help minimize your long-term storage costs.

Benefits of AWS Storage Gateway

• Scalability: AWS Storage Gateway scales effortlessly to accommodate changing data storage needs, allowing businesses to expand their storage capacity as required.

• Cost-Effectiveness: By leveraging AWS's pay-as-you-go pricing model, businesses can optimize storage costs and eliminate the need for upfront hardware investments.

• Data Security: With robust encryption and access controls, AWS Storage Gateway ensures the security and integrity of data during transit and at rest, providing peace of mind to businesses.

• Seamless Integration: AWS Storage Gateway seamlessly integrates with existing on-premises environments and AWS services, enabling smooth data transfer and management across hybrid cloud architectures.

• Fully Managed Cache: The local gateway appliance maintains a cache of recently written or read data so your applications can have low-latency access to data that is stored durably in AWS. The gateways use a read-through and write-back cache, committing data locally, acknowledging the write operations, and then asynchronously copying data to AWS, reducing application la

Use Cases of AWS Storage Gateway

• Hybrid cloud workflows store: File data as objects using data generated by on-premises applications for processing by AWS services such as machine learning or big data analytics.

• Migrate application data to EBS: Use a snapshot of your on-premises volumes to recreate the data on EBS and use with Amazon EC2-based applications.

• Back up data to the cloud: Provide cloud-based backup for on-premises files and database applications for low-cost, virtually unlimited scale.

Conclusion

As businesses navigate the complexities of modern data storage and management, AWS Storage Gateway emerges as a versatile solution to bridge the gap between on-premises environments and the cloud. With its diverse array of gateway options and seamless integration capabilities, AWS Storage Gateway empowers organizations to embrace the benefits of cloud storage while maintaining flexibility, scalability, and cost-efficiency. Embrace the power of AWS Storage Gateway and unlock new possibilities in data storage and management for your business.

Thank you for the read. I appreciate your time.

Follow for more Azure and AWS Content. Happy Learning!

Regards,
Jineshkumar Patel

In Amazon S3, objects are owned by the AWS account that was used to upload the object to a bucket. By default, the owner of a bucket has full control over the bucket, and every objects (i.e., a files) stored in a bucket. Including the ability to delete it and modify its access controls. Moreover, the owner of a bucket can grant other AWS accounts or users permission to access and manage the objects in the bucket.

This is important because it allows the owner to manage the security of their objects and ensure that only authorized users have access to them.

For example, consider a scenario where an AWS user named Alice has an S3 bucket called "unique-bucket-name" and she has stored several objects in it. Alice is the owner of these objects, so she has the ability to manage their access controls, delete them, and take other actions as needed when she is logged in into AWS Console using her Access. If another AWS user named Bob tries to access one of Alice's objects without permission, Alice can use the access controls on the object to prevent Bob from accessing it.

Controlling ownership of objects and disabling ACLs for your bucket

ACLs Disabled (Recommended)

• When ACLs are disabled and bucket owner enforcement is enabled (which is recommended), the bucket owner automatically becomes the owner of every object in the bucket and has full control over them.

• With this setting, ACLs no longer determine access control for data in the S3 bucket. Access control policies are used instead to define who can access the bucket and its contents.

ACL is disabled. Object Ownership is handled by Bucket Policies. "Edit" Grayed out.

How Bucket Policy work? (Important)

Amazon S3 offers two types of policies, bucket, and user policies, which allow or deny permissions to different resources. A policy consists of the following elements,

Resources – Resources refer to the Amazon S3 entities to that permissions can be granted or denied to. i.e. Buckets, objects, access points, and jobs

Actions – Actions are the operations that can be performed on Amazon S3 resources. allow or deny.

Effect – Effect determines whether a user is allowed or denied the action requested. If you do not explicitly grant access to (allow) a resource, access is implicitly denied.

Principal – Principal is the entity or user that is being granted or denied permission to access the resources or perform the actions.

Condition – Conditions specify the circumstances under which the policy is in effect and can be used to restrict access based on factors such as time or IP address.

For example, The following example bucket policy allows Dave, a user in account Account-ID, s3:GetObject, s3:GetBucketLocation, and s3:ListBucket Amazon S3 permissions on the awsexamplebucket1 bucket.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "Only allow writes to my bucket with bucket owner full control",
         "Effect": "Allow",
         "Principal": {
            "AWS": [
               "arn:aws:iam::111122223333:user/ExampleUser"
            ]
         },
         "Action": [
            "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",
         "Condition": {
            "StringEquals": {
               "s3:x-amz-acl": "bucket-owner-full-control"
            }
         }
      }
   ]
}

ACLs enabled

• Bucket owner preferred – The bucket owner owns and has full control over new objects that other accounts write to the bucket with the bucket-owner-full-control canned ACL.

• Object writer (default) – The AWS account that uploads an object owns the object, has full control over it, and can grant other users access to it through ACLs.

Object Ownership - Bucket Owner Preferred with ACL Enabled

Object Ownership - Object Writer with ACL Enabled

This represents How editing ACL for list/read/write as per the requirement for Objects and Bucket ACL settings and enforce to related grantees for access Including other AWS Account.

Requiring the bucket-owner-full-control canned ACL for Amazon S3 PUT operations (in Object ownership - bucket owner preferred)

When you enable the bucket owner preferred setting for Object Ownership, Bucket Owner have full control over the new objects that other accounts write to your bucket. If they use the "bucket-owner-full-control" canned ACL, you will be the owner of those objects. But, if they don't use this canned ACL, they will still have full control access. To prevent this, you can create a bucket policy that only allows writes with the "bucket-owner-full-control" canned ACL.

Example,
The following bucket policy specifies that account 111122223333 can upload objects to DOC-EXAMPLE-BUCKET only when the object's ACL is set to bucket-owner-full-control. Be sure to replace 111122223333 with your account and DOC-EXAMPLE-BUCKET with the name of your bucket.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "Only allow writes to my bucket with bucket owner full control",
         "Effect": "Allow",
         "Principal": {
            "AWS": [
               "arn:aws:iam::111122223333:user/ExampleUser"
            ]
         },
         "Action": [
            "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",
         "Condition": {
            "StringEquals": {
               "s3:x-amz-acl": "bucket-owner-full-control"
            }
         }
      }
   ]
}

This table shows the impact that each Object Ownership setting has on ACLs, objects, object ownership, and object uploads.

If you use S3 Versioning, the bucket owner owns and has full control over all object versions in your bucket. Applying the bucket owner enforced setting does not add a new version of an object.

Example, Using the AWS Command Line Interface (AWS CLI) s3api put-object operation includes the bucket-owner-full-control canned ACL, the object can be uploaded to a bucket with disabled ACLs.

aws s3api put-object --bucket DOC-EXAMPLE-BUCKET --key key-name --body path-to-file --acl bucket-owner-full-control

Changes introduced by disabling ACLs

References to learn more,

1. Controlling ownership of Objects and Disabling ACLs for your bucket.

2. Disabling CLs for all new Buckets and enforcing Object Ownership.

3. Heads-Up: Amazon S3 Security Changes Are Coming in April of 2023

4. Policies and Permissions in Amazon S3

Upcoming changes starting April 2023

"Bucket owner enforced setting" will be enabled for newly created buckets, making bucket ACLs and object ACLs ineffective, and ensuring that the bucket owner is the object owner no matter who uploads the object."

A subsequent attempt to set a bucket policy or an access point policy that grants public access will be rejected with a 403 Access Denied error.

"S3 Block Public Access" – All four of the bucket-level settings will be enabled for newly created buckets.

The Bucket owner-enforced setting will soon be enabled, which means that bucket and object ACLs will no longer be effective. This change is intended to ensure that the bucket owner is always the object owner, regardless of who uploads the object. If you wish to enable ACLs for a bucket, you will need to set the ObjectOwnership parameter to ObjectWriter in your CreateBucket request or call DeleteBucketOwnershipControls after creating the bucket. Please note that s3:PutBucketOwnershipControls permission will be required in order to use the parameter or function.

I hope this blog provides an understanding of S3 Bucket Ownership, along with practical examples and screen capture references, to help you gain a better understanding of the topic. Also, some references to read and learn more.

Thank you for the read. Hope you like it. I appreciate your time.

Follow for more Azure and AWS Content. Happy Learning!

Regards,

Jineshkumar Patel

Amazon EFS Elastic Throughput

"Use this mode for workloads with unpredictable I/O. With Elastic mode, your throughput scales automatically and you only pay for what you use".

Amazon EFS (Elastic File System) is a service that provides fully elastic and serverless file storage. While creating Amazon EFS (Elastic File System), a New performance Setting is been introduced in Throughput Mode as Amazon EFS Elastic Throughput Mode.

It is a new pay-as-you-use throughput mode for Amazon EFS that provides applications with as much throughput as they need. This mode is designed for dynamic and unpredictable workloads with difficult-to-forecast performance requirements. When enabled, Elastic Throughput actively manages file system performance and prevents over-payment for idle resources, ensuring optimal performance for applications.

Enabling Elastic Throughput eliminates the need to specify or provision throughput capacity, as Amazon EFS automatically delivers the required throughput performance. Customers only pay for the amount of data read or written.

With Elastic Throughput, Amazon EFS simplifies and extends its performance capabilities, allowing customers to use it for a wider range of file workloads. Amazon EFS is well suited for a variety of use cases, such as analytics, data science, machine learning, CI/CD tools, content management, web serving, and SaaS applications. Amazon EFS Elastic Throughput is available in all regions except for the AWS China regions.

Failover Controls for Amazon S3 Multi-Region Access Points

Amazon S3 Multi-Region Access Points provide a global endpoint for accessing S3 buckets in multiple AWS regions. This feature allows for the creation of multi-region applications using the same simple architecture as in a single region.

The Multi-Region Access Point uses AWS Global Accelerator to monitor network congestion and connectivity and routes traffic to the closest copy of the data.

If connectivity is lost between a client and a bucket in a particular region, the Multi-Region Access Point will automatically route traffic to the closest bucket (synchronized via S3 Replication) in another region.

Failover controls for Multi-Region Access Points allow users to shift S3 data access request traffic routed through a Multi-Region Access Point to an alternate AWS region within minutes. This allows users to test and build highly available applications for business continuity.

The existing Multi-Region Access Point model treats all regions as active and can send traffic to any of them. The new model introduced at AWS re:Invent allows users to designate regions as either active or passive. Buckets in active regions receive traffic (GET, PUT, and other requests) from the Multi-Region Access Point, while buckets in passive regions do not. Amazon S3 Cross-Region Replication operates regardless of the active or passive status of a region concerning a particular Multi-Region Access Point.

Amazon S3 > Multi-Region Access Points > Create Multi-Region Access Point

"By default, my new Multi-Region Access Point routes traffic to all of the buckets, and behaves as it did before we launched this new feature. However, I can now exercise control over routing and failover. I click on the Multi-Region Access Point, and on the Replication and failover tab (which used to be just a Replication tab). The map now allows me to see my replication rules and my failover status:" (Source from Author: Jeff Barr )

The tab for failover configuration (NEW): (source from Author: Jeff Barr )

"To change my failover configuration, I select the buckets of interest and click Edit failover configuration. My application runs in the Asia Pacific (Tokyo) Region and makes use of a bucket there, so I leave the Tokyo Region active and make the others passive:" (Source from Author: Jeff Barr )

New for AWS Backup – Protect and Restore Your CloudFormation Stacks

AWS Backup is a service offered by Amazon Web Services (AWS) that provides automated backups for AWS resources. The new feature allows users to protect and restore their CloudFormation stacks using AWS Backup.

CloudFormation is a service that allows users to create and manage infrastructure as code, and the ability to protect and restore these stacks using AWS Backup provides added security and resilience for users of the service.

When you use AWS Backup to protect your CloudFormation stacks, it creates a backup of all the stateful resources in the stack, such as Amazon Elastic Compute Cloud (Amazon EC2) instances and Amazon Elastic Block Store (Amazon EBS) volumes, at the same time. The backup also includes the stateless resources in the stack, such as IAM roles and Amazon VPC security groups, allowing you to restore the entire stack to its previous state if necessary.

Above Images source from here.

There is no additional cost for the stateless resources backed up and restored by AWS Backup. You only pay for the stateful resources such as databases, storage volumes, or file systems. For more information, see AWS Backup pricing.

Amazon Redshift Supported in AWS Backup

Amazon Redshift is a fully managed, cloud-based data warehousing service offered by AWS. AWS Backup allows users to automate the process of creating backups of their Amazon Redshift clusters, and provides options for scheduling these backups and setting retention policies.
AWS Backup > Settings > Configure Resources > Redshift - new

There is no additional cost for using AWS Backup compared to the native snapshot capability of Amazon Redshift. Overall costs depend on the amount of storage and retention you choose. For more information, see AWS Backup pricing.

Automated in-AWS Failback for AWS Elastic Disaster Recovery

Automated in-AWS failback for AWS Elastic Disaster Recovery (AWS EDR) is a feature that allows users to automatically fail back to their original primary AWS Region after a disaster recovery event. When a disaster occurs, AWS EDR automatically fails over to a specified secondary AWS Region to keep applications running. After the disaster has been resolved and the original primary AWS Region is available again, the automated in-AWS failback feature automatically fails back to the primary AWS Region, allowing users to resume normal operation in their primary environment. This helps to simplify the disaster recovery process and reduce downtime for applications.

AWS Elastic Disaster Recovery (AWS EDR) is a service that helps users to protect their applications and data from disasters by maintaining a constant replication posture for their operating systems, applications, and databases.

AWS EDR automatically fails over to a specified secondary AWS Region in the event of a disaster, allowing users to keep their applications running. The service now supports in-AWS failback, which allows users to automatically fail back to their primary AWS Region after the disaster has been resolved, as well as existing support for non-disruptive recovery drills and on-premises failback. This helps users to simplify the disaster recovery process and reduce downtime for their applications.

Failover vs. Failback

Failover is switching the running application to another AZ or Region.

Failback is the process of returning the application to the original AZ or Region.

Three Strages : 1. Preparation 2. Failover 3. Failback 4. Cleanup

Let’s assume an incident occurs with an in-AWS application, so we initiate a failover to another AWS Region. When the issue has been resolved, we want to fail back to the original Region. The following animation illustrates the failover and failback processes. Source here.

Learn more about in-AWS failback with Elastic Disaster Recovery here.

Those are Top Announcement regarding Storage in re:Invent this year(2022)

There are tons more announcements and exciting features announce throughout the year but mainly during re:invent. To keep up to date with AWS news, follow AWS News Blog.

Thank you for the read. Hope you like it.
I appreciate your time.

Follow for more Azure and AWS Content. Happy Learning!

Regards,
Jineshkumar Patel

Amazon S3 (Simple Storage Service) is a cloud storage service provided by Amazon Web Services (AWS). S3 buckets are used to store and manage data in the cloud. S3 provides a simple and highly scalable way to store and retrieve large amounts of data from anywhere on the internet. This makes it a useful service for a wide range of applications, including data storage, backup and recovery, and disaster recovery.

S3 bucket security is very important for sensitive and confidential data stored in those storage buckets. Securing the data stored in S3 buckets from unauthorized access, accidental deletion, and other potential threats. Therefore, it is essential to ensure that the data stored in S3 is properly secured to protect against unauthorized access and other potential security risks.

Tips for securing your S3 buckets

• Enable versioning to keep track of changes to your data

• Use encryption to protect your data while in transit and at rest

  • Server-Side Encryption – Request Amazon S3 to encrypt your object before saving it on disks in its data centers and then decrypt it when you download the objects.

    • To configure server-side encryption, see Specifying server-side encryption with AWS KMS (SSE-KMS) or Specifying Amazon S3 encryption.

  • Client-Side Encryption – Encrypt data client-side and upload the encrypted data to Amazon S3. In this case, you manage the encryption process, the encryption keys, and related tools.

    • To configure client-side encryption, see Protecting data using client-side encryption.

      

  • Limit access to your S3 buckets using Bucket policies and IAM policies

    

    Bucket Policy

    • AWS has given an easy way to generate S3 Bucket Policy as shown here.

      

      AWS Console > S3 > Select Bucket > Permissions > Bucket Policy > Edit

      Sample Bucket Policy

      

      "Principles" > "Effect" > "Actions" > "Resource" : S3 Bucket / Object / *

      Principles: Specifies whom the statement covers (Acc/Role/User/Srvs)
      Effect: Specifies whether the policy results in "deny" or "allow"
      Actions: Specifies the actions that is/are being denied or allowed.

      Resource: Specifies which AWS Resource the Policy Applies to. S3 Bucket

      Now, this Bucket Policy can be applied to a Bucket as needed.

      IAM Policy

      

      /AWS Console > Security & Identity > IAM > Create Policy

      Sample IAM Policy

      

      "Effect" -- > "Action" --> "Resource" : S3 Bucket / Object / * (all)

      Effect: Specifies whether the policy results in "deny" or "allow"
      Actions: Specifies the actions that is/are being denied or allowed.

      Resource: Specifies which AWS Resource the Policy Applies to. S3 Bucket / Object / *

      Now, this IAM Policy can be applied to IAM User or IAM Role as needed.

      Additionally, We can add a Condition element to these IAM/Bucket Policies

      "Condition" : {
      "{condition-operator}" : { "{condition-key}" : "{condition-value}" }}
      "Condition": {
      "StringEquals": {"aws:PrincipalTag/job-category": "iamuser-admin"}

      

      Policy Example with Condition :

      

  • AWS S3 Block Public Access

    • AWS applies protection by default against all accidental public access by Blocking all Public Access as a default setting while creating a Bucket.

    • This provides protection to permissions from ACL, Bucket Policy, or both.

      At account Level

      

      At Bucket Level:

      

Monitor and audit access to your S3 buckets

• Access Analyzer for S3

  • It analyzes permissions for all Buckets in a particular AWS Region.

  • And Provides a Dashboard to show findings from public buckets and buckets shared with external users or accounts.

• Access Analyzer reviews all Bucket Policies, Access Control Lists, Access Point Policy, and Block all Public Access Settings to produce findings.

Best practices for S3 bucket security

• S3 Bucket Names are Globally UNIQUE (i.e. No two S3 buckets can have the same name Globally). Use unique and complex names for your S3 buckets to prevent unauthorized access.

  • To prevent Brut Force attacks and prevent Malicious Actor to find out your S3 Bucket Name, It is important to choose a name that is complex and not easily guessable to prevent unauthorized access to your bucket.

  • This means avoiding using simple names or common words, and instead using a combination of letters, numbers, and special characters to make the bucket name more difficult to guess.

• Use multi-factor authentication for your AWS account to add an extra layer of security

• Regularly review and update your IAM policies and bucket policies to ensure they are up to date.

  • Amazon S3 uses Identity and Access Management (IAM) policies and bucket policies to control access to your S3 buckets and the objects they contain. These policies specify who is allowed to access your S3 resources and what actions they are allowed to perform on them. It is important to regularly review and update these policies to ensure they are up to date and still reflect your desired access control settings.

  • For example, if you have added new users to your AWS account, you will need to update your IAM policies to grant them access to your S3 resources. Similarly, if you have changed the way your data is organized in S3, you may need to update your bucket policies to reflect the new structure. Regularly reviewing and updating your IAM and bucket policies will help ensure that only authorized users have access to your S3 resources and that they can only perform the actions you have explicitly allowed.

• Use Amazon S3 ACLs to control access to specific objects in your S3 buckets

  • In addition to using IAM policies and bucket policies to control access to your Amazon S3 resources, you can also use Access Control Lists (ACLs) to fine-tune access to specific objects within your S3 buckets. An S3 ACL is a set of rules that define who can access an individual object and what actions they are allowed to perform on it.

  • For example, you could use an S3 ACL to allow certain users to read an object but not write to it, or to allow public access to an object while preventing delete operations. This allows you to have more granular control over access to your S3 objects, enabling you to fine-tune your security settings as needed.

  • You can ensure that only authorized users have access to your S3 objects and that they can only perform the actions you have explicitly allowed.

• Keep Public Buckets in a Dedicated AWS Account.

  • This will prevent any possibilities of public access to your S3 Bucket Data on Private Buckets by enabling "Block All Public Access" at AWS Account Level as it's in a separate AWS Account and Public Buckets are in a separate AWS Account.

• Run and Review Access Analyzer for S3 to see any Access Vulnerabilities and Findings can be analyzed and corrected.

• Use Encryption at Rest for each S3 Bucket in your Account.

  • Use Amazon S3-managed Keys (SSE-S3) - Free for all Buckets at least.

  • There are other Options for Encryption like "AWS KMS keys (SSE-KMS) with Customer Managed Keys" and "AWS KMS keys (SSE-KMS) with AWS Managed Key" to choose from.

Conclusion

There are several ways to secure an S3 bucket and the data it contains, including the following:

• Use access control lists (ACLs) to specify which users and groups are allowed to access the bucket and the types of actions they can perform. This can help prevent unauthorized access to the bucket and its contents.

• Enable versioning on the bucket to protect against accidental deletion or overwriting of objects. This allows you to recover previous versions of objects if they are deleted or modified.

• Enable server-side encryption for the bucket to protect the data stored in it. This encrypts the data at rest, so that it cannot be read by unauthorized users.

• Use Amazon S3 bucket policies to define additional security controls for the bucket, such as requiring SSL/TLS for all connections to the bucket or restricting access to the bucket based on the source IP address of the request.

• Monitor and log access to the bucket to identify potential security issues and to track who is accessing the bucket and its contents. This can help you detect and respond to potential security threats.

  By implementing these security measures, you can help protect your S3 bucket and the data it contains from unauthorized access and other potential security risks.

Thank you for reading and/or following along with the Blog.

Happy Learning.

Like and Follow for more Azure and AWS Content.

Regards,

Jineshkumar Patel

This Blog explains frequently used concepts and commands for S3 CLI Tasks. It is widely used to save time, and customize and automate S3 Storage Tasks.

Let's Dive-In,

• Install AWS CLI by following up here, as per your OS and Processor.
• Now, it’s time to configure the AWS profile. Use “AWS configure” It will need Admin User's (AdministratorAccess) AWS Access Key ID: and AWS Secret Access Key:
  (find your credentials under AWS console > IAM > Users > Create/Choose your Admin User > Under Security Credentials tab > Create Access Key (Securely Save this .cve file) And put those key info in the prompt for "AWS configure" sign in on AWS CLI. Example,
  I am in, verified by "aws s3 ls" command which listed available S3 buckets with its creation date and time.

Now We will go to list all useful commands for S3 CLI which will be useful to save you time, and automate S3 Storage operations with customization options.

Note : Assume your local directory and Your unique S3 Bucket Names respectively in below examples and screenshots.

AWS S3 Copy : aws s3 cp

Copy local file or S3 object to another S3 bucket or locally

# Copying a local file to S3
aws s3 cp test.txt  s3://mybucket/test2.txt

# Copying a file from S3 to S3
aws s3 cp s3://mybucket/test.txt  s3://mybucket/test2.txt
# Copying an S3 object to a local file
aws s3 cp s3://mybucket/test.txt  G:\test2.txt
# Copying an S3 object from one bucket to another
aws s3 cp s3://mybucket/test.txt  s3://mybucket2/

# Setting the Access Control List (ACL) while copying S3 object
aws s3 cp s3://mybucket/test.txt s3://mybucket/test2.txt --acl public-read-write

# Recursively copying 
aws s3 cp s3://mybucket D:\testfolder --recursive
aws s3 cp D:\testfolder  s3://mybucket/ --recursive 
aws s3 cp s3://mybucket/ s3://mybucket2/ --recursive

AWS S3 List : aws s3 ls

List S3 objects and common prefixes under a prefix or all S3 buckets

# Lists all of the buckets owned by the user
aws s3 ls
# Recursively list objects in a bucket
aws s3 ls s3://mybucket --recursive
# Recursively list objects in a bucket with human-readable
and summarize option 
aws s3 ls s3://mybucket --recursive --human-readable --summarize

AWS S3 Move : aws s3 mv

Move local file or S3 object to another S3 bucket or locally

# Moves single/multi s3 object/s to a specified bucket 
- Changing the name of the target bucket file optionally. 
aws s3 mv G:\test.txt  s3://mybucket/test2.txt
# Move From local folder objects to move to s3 bucket 
aws s3 mv G:\testfolder  s3://mybucket/test2.txt
# Move all Objects From one S3 Bucket to another S3 Bucket 
aws s3 mv s3://s3clibucket-1  s3://s3clibucket-2  --recursive
# Moves all files but exclude some file with specific extension
aws s3 mv G:\testfolder  s3://mybucket/ --recursive --exclude "*.jpg"

Create an S3 buckets : aws s3 mb

mb (Make Bucket) command creates a bucket.

# Creates a bucket (Unique name in default region) 
aws s3 mb s3://myuniquenamebucket
# Create a s3 bucket in a specific Region
aws s3 mb s3://mybucket --region us-west-1

Generate pre-sign URL for S3 Object:

aws s3 presign

# s3 presign url (default 3600 seconds)
aws s3 presign s3://s3clibucket-1/test1.txt
aws s3 presign s3://s3clibucket-2/test2.txt --expires-in 60

To delete an S3 bucket: aws s3 rb

# Remove an empty bucket
aws s3 rb s3://mybucket
# Remove all the objects forcefully in the bucket and then remove
the bucket itself.
aws s3 rb s3://mybucket --force

To delete S3 object: aws s3 rm

# Delete an object from S3 Bucket
aws s3 rm s3://mybucket/test2.txt
# Recursively deletes all objects under a specified bucket
aws s3 rm s3://mybucket --recursive
# Deletes all objects but excluding some objects
aws s3 rm s3://mybucket/ --recursive --exclude "*.jpg"
aws s3 rm s3://mybucket/ --recursive --exclude "another/*"

S3 Static Website configuration for bucket:

aws s3 website

# Configures a bucket name as a static website
aws s3 website s3://my-bucket/ --index-document
index.html --error-document error.html

To Sync directories and S3 prefixes:

aws s3 sync

# Syncs objects with S3 bucket to local directory
aws s3 sync G:\Testfolder  s3://mybucket

# Syncs files between two buckets in different regions
aws s3 sync s3://s3clibucket-1 s3://s3clibucket-2 --source-region us-west-2 --region us-east-1

# Syncs objects under a two different buckets
aws s3 sync s3://s3clibucket-1 s3://s3clibucket-2

# Sync but exclude objects with specific extensionor directory
aws s3 sync G:\Testfolder  s3://s3clibucket-1 --exclude "*.jpg"
aws s3 sync s3://s3clibucket-1/  G:\TestFolder --exclude "*another/*"

With this blog, I hope it clears out the starter concepts with AWS S3 CLI Commands and encouraged you to try it out yourself.

Thank you for reading and/or following along with the Blog.

Happy Learning.

Like and Follow for more Azure and AWS Content.

Regards,
Jineshkumar Patel

Reasons why we may need to copy our data to reside in more than one AWS region

1. Business Requirement or Compliance Reasons
2. Disaster Recovery Policy and in need to enhance the durability of Data
3. Need Second Copy in different S3 Storage Class / AWS Account / AWS Region for availability / latency or optimization reasons.
   We can achieve all of the above objectives and more with S3 cross-region replication.

Key points for S3 Cross Region Replication

• When an object is replicated into another AWS region by default, all of the objects, metadata, access control lists and object tags are also replicated and any further changes made to the source objects, metadata, access control list, or object tags also trigger replication to the destination.
• We can choose to replicate entire Buckets OR Objects under a shared prefix only OR at the object level using certain object tags only replication.
• Additionally, We have added controls to change the ownership of the replicated object to a different AWS account to protect against accidental or malicious actions if the source account is compromised.
• In 2020, AWS introduce, S3 Replication (multi-destination) : It is intended for customers that want to create and maintain multiple copies of their data in one or more AWS Regions. Amazon S3 Replication now gives usthe ability to replicate data from one source bucket to multiple destination buckets in the same, or different AWS Regions.
• We can also configure S3 to store replicated data in any S3 storage class including S3 glacier irrespective of the storage class of the source objects to meet compliance requirements.
• Versioning must be enabled on both the source and destination bucket.

How S3 replication works

How to set up S3 Bucket Replication and It's Options

1. Creating Two Buckets in S3
   Source Region Bucket : s3replicationsource-us-east-1
   Target Region Bucket : s3replicationtarget-us-west-1

1. Create a Bucket Replication Rule
   

Replication Rule Options :

• Choose the replication rule status to enable and the name of the replication rule.
• Scope to Filter by Prefix , Tag or All Objects to replicate in Source Bucket.
• Choose Destination Bucket. Option of Same Account or Different Account. I choose my previously created us-west-1 bucket in the same account.
  
• Create new IAM Role or use existing S3 Required permissions enabled IAM Role.
• Choose if We want to enable "Encryption" on replication
  
• We can Choose the Destination Bucket Storage Class
  More on S3 Storage Classes
• Choose Appropriate Additional Option and Save
  Options are for notification, monitoring, delete marker replication and modification sync related.
• One Time Popup Option to "Replicate Existing Objects" as a batch job when We save the Replication Rule.
  
• Our Replication Rule and Created and in Enabled State
  

From this point forward, any objects added to my Source bucket (s3replicationsource-us-east-1) will be replicated to my Destination Bucket (s3replicationtarget-us-west-1).
And Not only that, Any modification, edit , overwrite, delete will be replication to destination bucket as well.

Test :
Object "1.jpg" i uploaded to my Source Bucket (us-east-1) in Standard Storage Class is automatically Replicated to my Destination Bucket(us-west-1) in One-Zone Storage Class.

Source Bucket
Destination Bucket

Delete Operations on Objects in Source Bucket is also replicated.

Additionally, Amazon S3 can publish replication events as in Event Notification.
Amazon S3 sends event notifications when an object fails replication, exceeds the 15-minute threshold, replicates after the 15-minute threshold, and misses tracking by replication metrics.

In Source S3 Bucket Properties > Select Event Notification > Create Event Name and Type : Replication Event and Desitnation to SNS Topic

In this Blog, I showed how Amazon S3 replication is an easy, fully managed, a customizable and low-cost feature that replicates objects between buckets in Cross-Region (CRR) or Same-Region(SRR) Replication within same AWS Account or Separate AWS Account.
Anyone can use this solution to build a data redundancy capability to meet regulatory compliance, business continuity, and disaster recovery requirements.

Thank you for reading and following along with the Blog.

Happy Learning.

Like and Follow for more Azure and AWS Content.

Regards,
Jineshkumar Patel

That way, even if there are any security breaches or attacks on your company’s system, all of the information will be protected.

"Dance like nobody is watching. Encrypt like everyone is.”
-Werner Vogels - VP & CTO - Amazon.com

Is your Data Protected at-rest in S3 Buckets ?
Is your Data Encrypted in-transit to & from S3 Buckets ?

Not sure?
Don't Worry. This Blog will cover almost all aspects of S3 Bucket Data Protection and Encryption with Why and How to do so.

And if you are doing so or know few aspects of it, This blog will help you avoid common but critical pitfalls.

Data protection refers to protecting data while
In-transit (as it travels to and from Amazon S3) and
At rest (while it is stored on disks in Amazon S3 data centers)

Key Objects to Cover:

• Options for S3 Encryption: Server-Side(SSE) or Client-Side Encryption
• How to configure S3 Default Encryption ?
• Common Best Practices for Data Protection and Compliance
• Avoid Unnecessary Costs when Enabling SSE.
• Why S3 Bucket Policy is Important to Enforce Encryption
• Cost of Encryption

Options for protecting data "at-rest" in Amazon S3:

Server-Side Encryption – Request Amazon S3 to encrypt your object before saving it on disks in its data centers and then decrypt it when the Customer request to download the objects. To configure server-side encryption,
During Creation of a Bucket, Enable server-side encryption with

• AWS Key Management Service key (SSE-KMS)
  or
• Specify Amazon S3-managed keys (SSE-S3) or
• Customer-provided encryption keys (SSE-C)

Client-Side Encryption – Encrypt data client-side and upload the encrypted data to Amazon S3. In this case, Customer manage the encryption process, the encryption keys, and related tools. Client-side encryption is the act of encrypting data locally to ensure its security as it passes to the Amazon S3 service. The Amazon S3 service receives your encrypted data and it does not play a role in encrypting or decrypting it.
To enable client-side encryption, you have the following options:

• Use a key stored in AWS Key Management Service (AWS KMS).
• Use a key that you store within your application. (Not-Recommended) AWS's Note : Your client-side keys and your unencrypted data are never sent to AWS. It's important that you safely manage your encryption keys. If you lose them, you can't decrypt your data.

Setting default "server-side encryption" behavior for Amazon S3

With Amazon S3 default encryption, you can set the default encryption behavior for an S3 bucket so that all new objects are encrypted when they are stored in the bucket.
The objects are encrypted using server-side encryption with either Amazon S3-managed keys (SSE-S3) or AWS KMS keys stored in AWS Key Management Service (AWS KMS) (SSE-KMS).

S3 Default Encryption

When you configure your bucket to use default encryption with SSE-KMS, you can also enable S3 Bucket Keys to decrease request traffic from Amazon S3 to AWS Key Management Service (AWS KMS) and reduce the cost of encryption.

When Creating an S3 Bucket, there will be option to Enable SSE
Note: There are no additional fees for using server-side encryption with Amazon S3-managed keys (SSE-S3). However, requests to configure the default encryption feature incur standard Amazon S3 request charges. See S3 Pricing

Encrypting existing unencrypted objects

To encrypt your existing Amazon S3 objects, you can use Amazon S3 Batch Operations. You provide S3 Batch Operations with a list of objects to operate on, and Batch Operations calls the respective API to perform the specified operation.

You can use the Batch Operations Copy operation to copy existing unencrypted objects and write them back to the same bucket as encrypted objects. A single Batch Operations job can perform the specified operation on billions of objects. More on that here

I would love to perform this Batch Operation for S3 Encryption as a Hands-On Lab in upcoming Blogs. (Added in To-Do List)

Encrypting existing Amazon S3 objects with the AWS CLI

Simply, If you must encrypt all objects in your S3 bucket, you can run the following command: (More options on this here)

aws s3 cp s3://awsexamplebucket/ s3://awsexamplebucket/ --sse aws:kms --recursive

Is your Data Encrypted in-transit to & from S3 Buckets ?

• Data is more vulnerable when it’s in motion. To protect data in transit, companies should implement network security controls like firewalls and network access control. These will help secure the networks used to transmit information against malware attacks or intrusions.
• SSL/TLS uses both asymmetric and symmetric encryption to protect the confidentiality and integrity of data-in-transit. Both the client and server use HTTPS (SSL/TLS + HTTP) for their communication and can be used for File(Data) Transfer.
• TLS is a cryptographic protocol that provides end-to-end security of data sent between applications over the Internet. It should be noted that TLS does not secure data on end systems(When at-rest). It simply ensures the secure delivery of data over the Internet, avoiding possible eavesdropping and/or alteration of the content.

Why S3 Bucket Policy is Important to Enforce Encryption

• Bucket Policy can be set to prevent Users or Applications requests to Put Objects without Encryption or with different encryption method.
• Bucket Policy is Important with Default Encryption for the Bucket to make sure all the objects in the bucket comply with Certain Encryption Standard. (Making the Data Protection Officer Happy 🕵️😎😊🕵️‍♀️ for Compliance !! )
• In order to enforce object encryption on S3 Bucket, create an S3 bucket policy that denies any S3 Put request that does not include the x-amz-server-side-encryption header. There are two possible values for the x-amz-server-side-encryption header: AES256, which tells S3 to use S3-managed keys, and aws:kms, which tells S3 to use AWS KMS–managed keys.

the following bucket policy denies permissions to upload an object unless the request includes the x-amz-server-side-encryption header : AES256 to request server-side encryption:

{
  "Version": "2012-10-17",
  "Id": "PutObjectPolicy",
  "Statement": [
    {
      "Sid": "DenyIncorrectEncryptionHeader",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::awsexamplebucket1/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "AES256"
        }
      }
    }
  ]
}

the Above bucket policy denies the upload object (s3:PutObject) permission to everyone if the request does not include the x-amz-server-side-encryption header requesting server-side encryption with SSE-KMS.

OR

{
   "Version":"2012-10-17",
   "Id":"PutObjectPolicy",
   "Statement":[{
         "Sid":"DenyUnEncryptedObjectUploads",
         "Effect":"Deny",
         "Principal":"*",
         "Action":"s3:PutObject",
         "Resource":"arn:aws:s3:::DOC-EXAMPLE-BUCKET1/*",
         "Condition":{
            "StringNotEquals":{
               "s3:x-amz-server-side-encryption":"aws:kms"
            }
         }
      }
   ]
}

Both above Example Bucket Policy Enforcing Encryption to use "AES256" (SSE-S3) / aws:kms (Respectively) to Allow Put Objects to this Bucket.

Cost comes after Security

• Cost is also extremely important when dealing with Huge amount of Objects and their Buckets. And Enforcing S3 Security and Encryption comes with a Cost.
• Amazon S3 Bucket Keys reduce the request costs of Amazon S3 server-side encryption (SSE) with AWS Key Management Service (KMS) by up to 99% by decreasing the request traffic from S3 to KMS. With a few clicks in AWS Management Console and no changes to your client applications, you can configure your buckets to use an S3 Bucket Key for KMS-based encryption on new objects.

• Reduce the cost of S3 encryption

  When you configure default encryption to your bucket with SSE-KMS, you can also enable S3 Bucket Keys to decrease request traffic from Amazon S3 to AWS Key Management Service (AWS KMS) and reduce the cost of encryption. For more information, see Reducing the cost of SSE-KMS with Amazon S3 Bucket Keys.

Workloads that access millions or billions of objects encrypted with SSE-KMS can generate large volumes of requests to AWS KMS.

• When you use SSE-KMS to protect your data without an S3 Bucket Key, Amazon S3 uses an individual AWS KMS data key for every object. It makes a call to AWS KMS every time a request is made against a KMS-encrypted object.
• When you configure your bucket to use an S3 Bucket Key for SSE-KMS, AWS KMS generates a bucket-level key that is used to create unique data keys for new objects that you add to the bucket. This S3 Bucket Key is used for a time-limited period within Amazon S3, reducing the need for Amazon S3 to make requests to AWS KMS to complete encryption operations.
  This reduces traffic from S3 to AWS KMS, allowing to access AWS KMS-encrypted objects in S3 at a fraction of Cost compared to the previous approach(without an S3 Bucket Key).

  One Magic Trick that can save 99.8% on AWS S3 KMS charges

  

Let me Summarise what I explained in this Blog regarding S3 Encryption.

• Server-Side Encryption for S3 Buckets is Must for Data Protection.
• Enable Default Encryption
• Enforce Bucket Policy to Use S3 Encryption for Put Object.
• Encrypt Unencrypted Objects in Bucket using Batch Job / AWS CLI
• Use S3 Bucket Keys for SSE-KMS to Reduce Cost for Encryption.
• Use Bucket Level Keys instead of Object Level Keys to reduce Cost on KMS requests.

Hope you have Enjoyed the Blog. Thank you for Reading.

Feel free to ask questions about how to encrypt data at rest on S3.

Happy Learning.

Like and Follow for more Azure and AWS Content.

Thank you,
Jineshkumar Patel