AWS DevOps Step-by-Step: Building Your First CI/CD Pipeline on AWS

Ready to automate your build, test, and deployment processes on the world’s leading cloud platform? This AWS DevOps step-by-step guide will walk you through creating a fundamental Continuous Integration and Continuous Deployment (CI/CD) pipeline using core AWS services.

Why AWS for DevOps?

• Integrated Toolchain: Services like CodeCommit, CodeBuild, CodeDeploy, and CodePipeline work seamlessly together.
• Scalability & Flexibility: Pay only for what you use and scale resources as needed.
• Infrastructure as Code (IaC): Manage your infrastructure programmatically using services like CloudFormation or CDK.
• Managed Services: Reduce operational overhead by letting AWS manage the underlying infrastructure for your DevOps tools.

What We’ll Build:

We’ll create a simple CI/CD pipeline that automatically triggers when code is pushed to a source repository. The pipeline will:

1. Source: Pull code from an AWS CodeCommit repository.
2. Build: Compile the code and run basic tests using AWS CodeBuild.
3. Deploy: Deploy the built application to an EC2 instance using AWS CodeDeploy.

(Diagram Suggestion: A simple visual flow chart showing CodeCommit -> CodePipeline -> CodeBuild -> CodeDeploy -> EC2)

Prerequisites

Before we begin, ensure you have:

1. An AWS Account: You’ll need access to the AWS Management Console.
2. IAM User with Permissions: Create an IAM user with sufficient permissions (e.g., PowerUserAccess for simplicity in this tutorial, but follow the principle of least privilege in production). Configure your AWS CLI locally if you prefer command-line interaction.
3. A Sample Application: A simple web application (e.g., Node.js/Express, Python/Flask, or even just static HTML/CSS/JS) ready to be committed to source control.
4. Git Installed: You’ll need Git installed on your local machine.
5. (Optional but Recommended): An EC2 instance launched and configured with the CodeDeploy agent installed. This will be our deployment target. You can find instructions in the AWS documentation for installing the CodeDeploy agent.

Step 1: Setting Up Source Control with AWS CodeCommit

What is CodeCommit? AWS CodeCommit is a fully-managed source control service that hosts secure Git-based repositories.

Action:

1. Navigate to CodeCommit: Go to the AWS Management Console and search for “CodeCommit”.
2. Create Repository: Click “Create repository”. Give it a name (e.g., my-devops-app) and an optional description. Click “Create”.
3. Configure Git Credentials: Follow the instructions provided by AWS to set up HTTPS Git credentials for AWS CodeCommit or configure SSH access for your IAM user. This allows your local Git client to securely connect.
4. Clone the Repository: Copy the HTTPS or SSH clone URL provided on the repository page. Open your terminal/command prompt and run:

git clone <your-repository-clone-url>
cd my-devops-app

1. Add Your Application Code: Copy your sample application files into this new local directory.
2. Commit and Push:

git add . 
git commit -m "Initial commit of sample application" 
git push origin main # Or master, depending on your default branch name

You now have your application code stored securely in AWS CodeCommit!

Step 2: Building and Testing with AWS CodeBuild

What is CodeBuild? AWS CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces software packages ready for deployment.

Action:

1. Create a buildspec.yml file: This file tells CodeBuild how to build your project. Create a file named buildspec.yml in the root of your repository with content similar to this (adjust based on your application’s stack):

version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: 18 # Or python: 3.9, java: corretto11, etc.
    commands:
      - echo Installing dependencies...
      # Example for Node.js: npm install
      # Example for Python: pip install -r requirements.txt
  build:
    commands:
      - echo Build started on `date`
      # Example for Node.js: npm run build (if applicable)
      # Example: Run unit tests here
      # Example: echo "No build commands needed for static site"
  post_build:
    commands:
      - echo Build completed on `date`
      # Example: Package necessary files for deployment
artifacts:
  files:
    - '**/*' # Package all files for deployment (adjust as needed)
  # If you built artifacts into a specific directory (e.g., 'dist' or 'build'):
  # base-directory: 'dist'
  # files:
  #  - '**/*'

1. Commit and Push buildspec.yml:

git add buildspec.yml
git commit -m "Add buildspec.yml for CodeBuild"
git push origin main

1. Navigate to CodeBuild: Go to the AWS Management Console and search for “CodeBuild”.
2. Create Build Project: Click “Create build project”.
   • Project name: my-app-build
   • Source: Select “AWS CodeCommit“, choose your repository (my-devops-app), and select the branch (main or master).
   • Environment: Choose “Managed image”, select an Operating System (e.g., Amazon Linux 2), a Runtime (e.g., Standard), and an Image version appropriate for your buildspec.yml (e.g., aws/codebuild/standard:5.0).
   • Buildspec: Keep “Use a buildspec file” selected (CodeBuild will automatically look for buildspec.yml in your repo root).
   • Artifacts: Configure where build artifacts should be stored (e.g., an S3 bucket – CodeBuild can create one for you or you can choose an existing one). Choose “S3” as the type. Select “Zip” for packaging if deploying to EC2 via CodeDeploy.
   • Service Role: Allow CodeBuild to create a new service role or select an existing one with necessary permissions (access CodeCommit, S3, CloudWatch Logs).
3. Create Build Project: Click “Create build project”.
4. (Optional) Start Build Manually: You can optionally click “Start build” to test if your CodeBuild project works correctly with your buildspec.yml.

Your code can now be automatically built and tested using AWS CodeBuild.

Step 3: Deploying to EC2 with AWS CodeDeploy

What is CodeDeploy? AWS CodeDeploy is a service that automates application deployments to various compute services such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers.

Action:

1. Create appspec.yml file: This file tells CodeDeploy how and where to deploy your application on the target instance(s). Create a file named appspec.yml in the root of your repository:

version: 0.0
os: linux # Or windows
files:
  - source: / # The root of your build artifact
    destination: /var/www/html # Destination directory on the EC2 instance (adjust as needed)
    # For Windows: destination: c:\inetpub\wwwroot
hooks:
  # These are lifecycle event hooks where you can run scripts
  # BeforeInstall:
  #   - location: scripts/install_dependencies.sh
  #     timeout: 300
  #     runas: root
  ApplicationStart:
    - location: scripts/start_server.sh # Example script to start your app
      timeout: 300
      runas: root
  # ValidateService:
  #  - location: scripts/validate_service.sh
  #    timeout: 300
  #    runas: root

1. (Optional) Create a scripts directory in your repo root and add the scripts referenced in the hooks section (e.g., start_server.sh). Make sure these scripts are executable (chmod +x scripts/*.sh).
2. Commit and Push appspec.yml (and scripts):

git add appspec.yml scripts/
git commit -m "Add appspec.yml and deployment scripts for CodeDeploy"
git push origin main

1. Navigate to CodeDeploy: Go to the AWS Management Console, search for “CodeDeploy”, and click “Applications” under “Deploy”.
2. Create Application:
   • Click “Create application”.
   • Application name: my-devops-application
   • Compute platform: Select “EC2/On-premises”.
   • Click “Create application”.
3. Create Deployment Group: Inside your newly created application, click “Create deployment group”.
   • Deployment group name: my-app-prod-deployment-group
   • Service Role: Create or select an IAM role that grants CodeDeploy permissions to access your EC2 instances (e.g., permissions to read tags, interact with EC2).
   • Deployment type: Choose “In-place” (updates existing instances) or “Blue/green” (creates new instances and shifts traffic – more advanced). Select “In-place” for this tutorial.
   • Environment configuration: Select “Amazon EC2 instances”. Use tags to identify the target EC2 instance(s) you prepared earlier (e.g., Tag Key: Environment, Value: Dev). Ensure the CodeDeploy agent is installed and running on these instances.
   • Deployment settings: Choose a deployment configuration (e.g., CodeDeployDefault.OneAtATime).
   • Load Balancer: Disable if you’re not using one for this simple setup.
   • Click “Create deployment group”.

You now have AWS CodeDeploy configured to deploy your application artifacts.

Step 4: Orchestrating with AWS CodePipeline

What is CodePipeline? AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.

Action:

1. Navigate to CodePipeline: Go to the AWS Management Console and search for “CodePipeline”.
2. Create Pipeline: Click “Create pipeline”.
   • Pipeline name: my-application-pipeline
   • Service role: Choose “New service role” (let CodePipeline create one) or select an existing compatible role. Ensure “Allow AWS CodePipeline to create a Service Role…” is checked if creating a new one.
   • Artifact store: Choose “Default location” or specify a custom S3 bucket for pipeline artifacts.
   • Click “Next”.
3. Add source stage:
   • Source provider: Select “AWS CodeCommit”.
   • Repository name: Choose my-devops-app.
   • Branch name: Select main (or master).
   • Change detection options: Keep “AWS CodePipeline” (default, uses CloudWatch Events for triggers).
   • Output artifact format: Choose “CodePipeline default”.
   • Click “Next”.
4. Add build stage:
   • Build provider: Select “AWS CodeBuild”.
   • Region: Ensure it’s the same region as your CodeBuild project.
   • Project name: Select the my-app-build project you created earlier.
   • Build type: Select “Single build”.
   • Click “Next”.
5. Add deploy stage:
   • Deploy provider: Select “AWS CodeDeploy”.
   • Region: Ensure it’s the same region as your CodeDeploy application.
   • Application name: Select my-devops-application.
   • Deployment group: Select my-app-prod-deployment-group.
   • Click “Next”.
6. Review and Create: Review all the settings and click “Create pipeline”.

Your pipeline will automatically trigger for the first time. You can watch the progress through the different stages. Now, any push to the main branch in your CodeCommit repository will automatically trigger this AWS CodePipeline!

Step 5: (Briefly) Infrastructure as Code (IaC)

While we manually created resources via the Console, in a real-world AWS DevOps setup, you’d use Infrastructure as Code (IaC) tools like AWS CloudFormation or the AWS Cloud Development Kit (CDK).

Why IaC?

• Consistency: Define infrastructure reliably every time.
• Version Control: Track changes to your infrastructure like code.
• Automation: Spin up or tear down entire environments automatically.

You could define your CodeCommit repo, CodeBuild project, CodeDeploy application/group, and even the CodePipeline itself in a CloudFormation template or CDK script. This is a crucial next step for mature DevOps practices.

Step 6: Monitoring with AWS CloudWatch

Visibility is key in DevOps. AWS CloudWatch provides monitoring and observability for your AWS resources and applications.

• CodeBuild/CodeDeploy Logs: Build and deployment logs are automatically sent to CloudWatch Logs, helping you troubleshoot failures.
• Pipeline Execution History: CodePipeline provides a visual history of executions, showing successes and failures at each stage.
• Metrics & Alarms: Monitor EC2 instance metrics (CPU, Memory, Network) and set CloudWatch Alarms to notify you of issues.

Regularly check CloudWatch Logs and Metrics to understand your pipeline’s health and application performance.

Conclusion: Your AWS DevOps Journey Begins

Congratulations! You’ve successfully built a basic, yet functional, CI/CD pipeline using core AWS DevOps services. By automating your source, build, and deploy stages with CodeCommit, CodeBuild, CodeDeploy, and CodePipeline, you’ve taken a significant step towards faster, more reliable software delivery.

This is just the beginning. Explore further:

• Advanced Deployment Strategies: Blue/Green deployments with CodeDeploy.
• Containerization: Integrate with Amazon ECS or EKS.
• Serverless: Deploy AWS Lambda functions via CodePipeline.
• Security: Integrate security scanning tools (e.g., AWS CodeGuru, static analysis tools in CodeBuild).
• Infrastructure as Code: Define your entire pipeline and infrastructure using CloudFormation or CDK.

Embracing DevOps on AWS empowers your teams to innovate faster and operate more efficiently. Keep experimenting and building!

What are your favorite AWS DevOps tools or practices? Share your thoughts in the comments below!