AWS Storage gateway and Terraform

So in my last job I was standing up a POC and part of it needed a storage gateway. Now I like to think of myself as semi-intelligent but bloody heck I cannot for the life of me get my head around CFT there is just too much JSON (and I like JSON A LOT more than XML). Step forward Terraform, what took me ~a week in CFT / AWS Console I was able to completely do in Terraform in 2 Days (1 of which was spent struggling with what is to be covered in this post).

Terraform while great, their docs are awful for one simple reason, their examples are isolated. Take the storage gateway for example it actually requires several separate resources, data and roles to make it work to just stand it up. Yet is there an end-to-end example shown? No.

So hopefully this post will save someone else the headache.


locals {
common_tags = "${map(
"app", "storage-gateway-example",
"app_env", "sandbox"
)}"
subnet = "subnet-123456"
vpcid = "vpc-123456"
keyname = "sandboxkey"
}

resource "aws_instance" "server" {
ami = "ami-0fbb2c4c2b6c88635"
instance_type = "m4.xlarge"
associate_public_ip_address = false
key_name = "${local.keyname}"
subnet_id = "${local.subnet}"
vpc_security_group_ids = ["${aws_security_group.storage-gateway-sg.id}"]
root_block_device {
volume_size = 80
volume_type = "gp2"
}
tags = "${merge(
local.common_tags,
map(
"Name", "storage-gateway-server"
)
)}"
}

resource "aws_ebs_volume" "cache-disk" {
availability_zone = "${aws_instance.server.availability_zone}"
size = 150
encrypted = true
type = "gp2"
tags = "${merge(
local.common_tags,
map(
"Name", "storage-gateway-server-cache-disk"
)
)}"
}

resource "aws_volume_attachment" "disk-attach" {
device_name = "/dev/xvdb"
volume_id = "${aws_ebs_volume.cache-disk.id}"
instance_id = "${aws_instance.server.id}"
force_detach = true
}

resource "aws_security_group" "storage-gateway-sg" {
name = "storage-gateway-sg"
vpc_id = "${local.vpcid}"
tags = "${merge(
local.common_tags,
map(
"Name", "storage-gateway-sg"
)
)}"
ingress {
protocol = "tcp"
from_port = 80
to_port = 80
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
protocol = "tcp"
from_port = 20048
to_port = 20048
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
protocol = "udp"
from_port = 20048
to_port = 20048
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
protocol = "tcp"
from_port = 22
to_port = 22
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
protocol = "tcp"
from_port = 111
to_port = 111
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
protocol = "udp"
from_port = 111
to_port = 111
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
protocol = "tcp"
from_port = 2049
to_port = 2049
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
protocol = "udp"
from_port = 2049
to_port = 2049
cidr_blocks = ["0.0.0.0/0"]
}
egress {
protocol = "tcp"
from_port = 0
to_port = 65535
cidr_blocks = ["0.0.0.0/0"]
}
egress {
protocol = "udp"
from_port = 0
to_port = 65535
cidr_blocks = ["0.0.0.0/0"]
}
}

resource "aws_storagegateway_gateway" "storage-gateway" {
gateway_ip_address = "${aws_instance.server.private_ip}"
gateway_name = "storage-gateway"
gateway_timezone = "GMT"
gateway_type = "FILE_S3"
}

data "aws_storagegateway_local_disk" "storage-gateway-data" {
disk_path = "${aws_volume_attachment.disk-attach.device_name}"
gateway_arn = "${aws_storagegateway_gateway.storage-gateway.arn}"
}

resource "aws_storagegateway_cache" "storage-gateway-cache" {
disk_id = "${data.aws_storagegateway_local_disk.storage-gateway-data.id}"
gateway_arn = "${aws_storagegateway_gateway.storage-gateway.arn}"
}

resource "aws_storagegateway_nfs_file_share" "nfs_share" {
client_list = ["0.0.0.0/0"]
gateway_arn = "${aws_storagegateway_gateway.storage-gateway.arn}"
location_arn = "${aws_s3_bucket.transfer-bucket.arn}"
role_arn = "${aws_iam_role.transfer-role.arn}"
}

resource "aws_iam_role" "transfer-role" {
name = "transfer-role"
assume_role_policy = < {
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "storagegateway.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}

resource "aws_iam_policy" "transfer-policy-sg" {
name = "transfer-policy-sg"
description = "Allows access to storage gateway"
policy = < {
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetAccelerateConfiguration",
"s3:GetBucketLocation",
"s3:GetBucketVersioning",
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:ListBucketMultipartUploads"
],
"Resource": "arn:aws:s3:::transfer-bucket",
"Effect": "Allow"
},
{
"Action": [
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:DeleteObjectVersion",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion",
"s3:ListMultipartUploadParts",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::transfer-bucket/*",
"Effect": "Allow"
}
]
}
EOF
}

resource "aws_iam_policy_attachment" "attach-policies" {
name = "storageGW-attachment"
roles = ["${aws_iam_role.transfer-role.name}"]
policy_arn = "${aws_iam_policy.transfer-policy-sg.arn}"
}

5 thoughts to “AWS Storage gateway and Terraform”

  1. Hello Rayn,

    Thank you for again for this post,
    I just want to say, While creating the Bucket S3 is acl “Private”

    Than following error with File sharing is occuring
    Error: error waiting for Storage Gateway NFS File Share creation: unexpected state ‘UNAVAILABLE’, wanted target ‘AVAILABLE’. last error: %!s()

    I changed it acl to “PublicRead” Than terraform creates filesharing resource too. May be you have other information regarding this error.
    best Regards
    Ram

  2. https://www.terraform.io/docs/providers/aws/r/storagegateway_gateway.html


    NOTE: One of activation_key or gateway_ip_address must be provided for resource creation (gateway activation). Neither is required for resource import. If using gateway_ip_address, Terraform must be able to make an HTTP (port 80) GET request to the specified IP address from where it is running.

    You are missing a route or you are using a private IP instead of a public one or firewall is blocking, would be my best guess.

  3. Hello Ryan,

    Seems Terraform sotragegateway resource when also tries to active and i am gettng error

    Error: error retrieving activation key from IP Address (172.31.18.11): error making HTTP request: Get http://172.31.18.11/?activationRegion=ap-southeast-1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

    I am researching to fix this if you know anything, would appreciate your help.
    Thank you

  4. You are correct. I cannot remember if I didn’t include it because S3 is easy to do or because i just forget. :)

  5. Hello Thank you for this post, I am looking for exactly this one.
    Just one think The aws_s3_bucket.aws_s3_bucket.transfer-bucket is not created by this code. right ?
    did you create it by console ?

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This blog is kept spam free by WP-SpamFree.