Introducing Terraform assist for Amazon OpenSearch Ingestion

At present, we’re launching Terraform assist for Amazon OpenSearch Ingestion. Terraform is an infrastructure as code (IaC) device that helps you construct, deploy, and handle cloud assets effectively. OpenSearch Ingestion is a completely managed, serverless knowledge collector that delivers real-time log, metric, and hint knowledge to Amazon OpenSearch Service domains and Amazon OpenSearch Serverless collections. On this publish, we clarify how you should utilize Terraform to deploy OpenSearch Ingestion pipelines. For example, we use an HTTP supply as enter and an Amazon OpenSearch Service area (Index) as output.

Answer overview

The steps on this publish deploy a publicly accessible OpenSearch Ingestion pipeline with Terraform, together with different supporting assets which are wanted for the pipeline to ingest knowledge into Amazon OpenSearch. We’ve got applied the Tutorial: Ingesting knowledge into a site utilizing Amazon OpenSearch Ingestion, utilizing Terraform.

We create the next assets with Terraform:

The pipeline that you simply create exposes an HTTP supply as enter and an Amazon OpenSearch sink to save lots of batches of occasions.

Stipulations

To observe the steps on this publish, you want the next:

• An lively AWS account.
• Terraform put in in your native machine. For extra info, see Set up Terraform.
• The required IAM permissions required to create the AWS assets utilizing Terraform.
• awscurl for sending HTTPS requests by the command line with AWS Sigv4 authentication. For directions on putting in this device, see the GitHub repo.

Create a listing

In Terraform, infrastructure is managed as code, known as a challenge. A Terraform challenge accommodates numerous Terraform configuration information, similar to fundamental.tf, supplier.tf, variables.tf, and output.df . Let’s create a listing on the server or machine that we are able to use to hook up with AWS providers utilizing the AWS Command Line Interface (AWS CLI):

mkdir osis-pipeline-terraform-example

Change to the listing.

cd osis-pipeline-terraform-example

Create the Terraform configuration

Create a file to outline the AWS assets.

Enter the next configuration in fundamental.tf and save your file:

terraform {
  required_providers {
    aws = {
      supply  = "hashicorp/aws"
      model = "~> 5.36"
    }
  }

  required_version = ">= 1.2.0"
}

supplier "aws" {
  area = "eu-central-1"
}

knowledge "aws_region" "present" {}
knowledge "aws_caller_identity" "present" {}
locals {
    account_id = knowledge.aws_caller_identity.present.account_id
}

output "ingest_endpoint_url" {
  worth = tolist(aws_osis_pipeline.instance.ingest_endpoint_urls)[0]
}

useful resource "aws_iam_role" "instance" {
  title = "exampleosisrole"
  assume_role_policy = jsonencode({
    Model = "2012-10-17"
    Assertion = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "osis-pipelines.amazonaws.com"
        }
      },
    ]
  })
}

useful resource "aws_opensearch_domain" "check" {
  domain_name           = "osi-example-domain"
  engine_version = "OpenSearch_2.7"
  cluster_config {
    instance_type = "r5.massive.search"
  }
  encrypt_at_rest {
    enabled = true
  }
  domain_endpoint_options {
    enforce_https       = true
    tls_security_policy = "Coverage-Min-TLS-1-2-2019-07"
  }
  node_to_node_encryption {
    enabled = true
  }
  ebs_options {
    ebs_enabled = true
    volume_size = 10
  }
 access_policies = <<EOF
{
  "Model": "2012-10-17",
  "Assertion": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "${aws_iam_role.example.arn}"
      },
      "Action": "es:*"
    }
  ]
}

EOF

}

useful resource "aws_iam_policy" "instance" {
  title = "osis_role_policy"
  description = "Coverage for OSIS pipeline function"
  coverage = jsonencode({
    Model = "2012-10-17",
    Assertion = [
        {
          Action = ["es:DescribeDomain"]
          Impact = "Enable"
          Useful resource = "arn:aws:es:${knowledge.aws_region.present.title}:${native.account_id}:area/*"
        },
        {
          Motion = ["es:ESHttp*"]
          Impact = "Enable"
          Useful resource = "arn:aws:es:${knowledge.aws_region.present.title}:${native.account_id}:area/osi-test-domain/*"
        }
    ]
})
}

useful resource "aws_iam_role_policy_attachment" "instance" {
  function       = aws_iam_role.instance.title
  policy_arn = aws_iam_policy.instance.arn
}

useful resource "aws_cloudwatch_log_group" "instance" {
  title = "/aws/vendedlogs/OpenSearchIngestion/example-pipeline"
  retention_in_days = 365
  tags = {
    Title = "AWS Weblog OSIS Pipeline Instance"
  }
}

useful resource "aws_osis_pipeline" "instance" {
  pipeline_name               = "example-pipeline"
  pipeline_configuration_body = <<-EOT
            model: "2"
            example-pipeline:
              supply:
                http:
                  path: "/test_ingestion_path"
              processor:
                - date:
                    from_time_received: true
                    vacation spot: "@timestamp"
              sink:
                - opensearch:
                    hosts: ["https://${aws_opensearch_domain.test.endpoint}"]
                    index: "application_logs"
                    aws:
                      sts_role_arn: "${aws_iam_role.instance.arn}"   
                      area: "${knowledge.aws_region.present.title}"
        EOT
  max_units                   = 1
  min_units                   = 1
  log_publishing_options {
    is_logging_enabled = true
    cloudwatch_log_destination {
      log_group = aws_cloudwatch_log_group.instance.title
    }
  }
  tags = {
    Title = "AWS Weblog OSIS Pipeline Instance"
  }
  }

Create the assets

Initialize the listing:

Evaluation the plan to see what assets might be created:

Apply the configuration and reply sure to run the plan:

The method would possibly take round 7–10 minutes to finish.

Take a look at the pipeline

After you create the assets, you need to see the ingest_endpoint_url output displayed. Copy this worth and export it in your setting variable:

export OSIS_PIPELINE_ENDPOINT_URL=<Exchange with worth copied>

Ship a pattern log with awscurl. Exchange the profile along with your acceptable AWS profile for credentials:

awscurl --service osis --region eu-central-1 -X POST -H "Content material-Sort: software/json" -d '[{"time":"2014-08-11T11:40:13+00:00","remote_addr":"122.226.223.69","status":"404","request":"GET http://www.k2proxy.com//hello.html HTTP/1.1","http_user_agent":"Mozilla/4.0 (compatible; WOW64; SLCC2;)"}]' https://$OSIS_PIPELINE_ENDPOINT_URL/test_ingestion_path

It is best to obtain a 200 OK as a response.

To confirm that the info was ingested within the OpenSearch Ingestion pipeline and saved within the OpenSearch, navigate to the OpenSearch and get its area endpoint. Exchange the <OPENSEARCH ENDPOINT URL> within the snippet given under and run it.

awscurl --service es --region eu-central-1 -X GET https://<OPENSEARCH ENDPOINT URL>/application_logs/_search | json_pp 

It is best to see the output as under:

Clear up

To destroy the assets you created, run the next command and reply sure when prompted:

The method would possibly take round 30–35 minutes to finish.

Conclusion

On this publish, we confirmed how you should utilize Terraform to deploy OpenSearch Ingestion pipelines. AWS provides numerous assets so that you can shortly begin constructing pipelines utilizing OpenSearch Ingestion and use Terraform to deploy them. You should use numerous built-in pipeline integrations to shortly ingest knowledge from Amazon DynamoDB, Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon Safety Lake, Fluent Bit, and plenty of extra. The next OpenSearch Ingestion blueprints let you construct knowledge pipelines with minimal configuration adjustments and handle them with ease utilizing Terraform. To be taught extra, take a look at the Terraform documentation for Amazon OpenSearch Ingestion.

In regards to the Authors

Rahul Sharma is a Technical Account Supervisor at Amazon Net Companies. He’s passionate in regards to the knowledge applied sciences that assist leverage knowledge as a strategic asset and is predicated out of NY city, New York.

Farhan Angullia is a Cloud Software Architect at AWS Skilled Companies, based mostly in Singapore. He primarily focuses on trendy functions with microservice software program patterns, and advocates for implementing strong CI/CD practices to optimize the software program supply lifecycle for purchasers. He enjoys contributing to the open supply Terraform ecosystem in his spare time.

Arjun Nambiar is a Product Supervisor with Amazon OpenSearch Service. He focusses on ingestion applied sciences that allow ingesting knowledge from all kinds of sources into Amazon OpenSearch Service at scale. Arjun is fascinated with massive scale distributed techniques and cloud-native applied sciences and is predicated out of Seattle, Washington.

Muthu Pitchaimani is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search functions and options. Muthu is within the subjects of networking and safety, and is predicated out of Austin, Texas.