Add filter for opensearch default packages

[swhite-oreilly]

This PR includes adds a filter for default Opensearch packages that were recently added by AWS. (We confirmed with AWS support that AWS created packages are the only ones that will have an ID prefix of G).

Testing

The following script adds a custom opensearch domain with packages. After running this script, you can confirm the default packages are now skipped and only user created packages will get removed.

# Create an OpenSearch domain
echo "Creating OpenSearch domain"
DOMAIN=`aws opensearch create-domain --domain-name mylogs --engine-version OpenSearch_1.2 --cluster-config  InstanceType=t3.small.search,InstanceCount=1 --ebs-options EBSEnabled=true,VolumeType=gp3,VolumeSize=100,Iops=3500,Throughput=125 --access-policies '{"Version": "2012-10-17", "Statement": [{"Action": "es:*", "Principal":"*","Effect": "Allow", "Condition": {"IpAddress":{"aws:SourceIp":["192.0.2.0/32"]}}}]}'`

# Extract the domain name using grep and awk
DOMAIN_NAME=$(echo "$DOMAIN" | grep -o '"DomainName": "[^"]*' | awk -F'"' '{print $4}')

# Print the extracted domain name
echo "The domain name is: $DOMAIN_NAME"

# Extract the ARN using grep and awk
DOMAIN_ARN=$(echo "$DOMAIN" | grep -o '"ARN": "[^"]*' | awk -F'"' '{print $4}')
echo "The domain ARN is: $DOMAIN_ARN"

# Create a file to upload to S3
echo "Generating file to upload to S3 for OpenSearch custom package"
cat <<EOF > synonyms.txt
danish, croissant, pastry
ice cream, gelato, frozen custard
sneaker, tennis shoe, running shoe
basketball shoe, hightop
EOF

# Generate a random string to use as a bucket name
RANDOM_STRING=$(openssl rand -hex 20)

# Create a S3 Bucket to store the file:
echo "Creating S3 bucket to store OpenSearch custom package file"
aws s3api create-bucket --bucket opensearch-packages-$RANDOM_STRING --region us-east-1

# Copy the provided generated synonym to the S3 bucket:
echo "Uploading file to S3 bucket"
aws s3 cp ./synonyms.txt s3://opensearch-packages-$RANDOM_STRING

# Creating an OpenSearch custom package (Note: This command will fail if the bucket and file do not exist)
echo "Creating OpenSearch custom package"
aws opensearch create-package --package-name test-package --package-type 'TXT-DICTIONARY' --package-source '{"S3BucketName": "'"opensearch-packages-$RANDOM_STRING"'", "S3Key": "synonyms.txt"}'


echo "Getting a subnet from the default VPC"
DEFAULT_VPC_ID=$(aws ec2 describe-vpcs --filters "Name=isDefault,Values=true" --query 'Vpcs[0].VpcId' --output text)

SUBNET_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$DEFAULT_VPC_ID" "Name=availability-zone,Values=us-east-1*" --query 'Subnets[0].SubnetId' --output text)
echo "Using subnet $SUBNET_ID"

# Get the security group ID
SECURITY_GROUP_ID=$(aws ec2 describe-security-groups --filters "Name=vpc-id,Values=$DEFAULT_VPC_ID" --query 'SecurityGroups[0].GroupId' --output text)
echo "Using security group $SECURITY_GROUP_ID"

# Create an Opensearch VPC Endpoint (Note: Using the default VPC appears to work intermittently, but creating a new VPC and then creating the VPC endpoint works consistently)
echo "Creating OpenSearch VPC Endpoint"
VPC_OPTIONS='{"SubnetIds": ["'"$SUBNET_ID"'"], "SecurityGroupIds": ["'"$SECURITY_GROUP_ID"'"]}'
echo "VPC options: $VPC_OPTIONS"

# Create the VPC endpoint
aws opensearch create-vpc-endpoint \
  --domain-arn "$DOMAIN_ARN" \
  --vpc-options "$VPC_OPTIONS"

[der-eismann]

Hey @swhite-oreilly, thanks for the quick fix! Highly appreciated 🙂