How to increase disk space on existing AWS EC2 Linux (Ubuntu) Instance without losing data

By Semir H.

Let’s say you have a Linux Amazon Web Services (AWS) EC2 Instance up and running and you start running out of space. I’ll show you how to quickly clone its disk (EBS volume) onto a bigger disk (also EBS), replace the smaller disk with the bigger one and boot off the bigger (new) disk. No need for any third party tools like Acronis or Ghost or even rsync.


If you find this tutorial useful, please consider making a donation to support future work like this:

Thank you :)

Create Snapshot of the original disk:

First thing we’ll do is to create a Snapshot of the original disk so that we can then create a new, bigger volume out of that Snapshot. There are a couple of things we have to note down first: the Instance number and then the volume attached to that Instance. The easiest way to fine the Instance number is in the Instances part of the AWS Management Console. It will start with “i” and be under the “Instance” column. Now navigate to the Volumes section (Elastic Block Store section) and locate the volume that is attached to your Instance. You can see the Instance number under the “Attachment Information” column. The first half of that string will be the Instance number. Please see the images below for an example.

Instance (in this case number is i-1920b74c):

Please also note which Zone your Instance is in. This will be important for later. In my case the Zone is “ap-southeast-1b” in the Singapore Region. The Zone can be seen by selecting the Instance and looking under its Description tab (bottom).

Volume (note the Instance number):

As you’ll notice in the examples above, the size of the original Volume (EBS Disk) is 8 GB. I’ll increase that to 25 GB.

Creating a Snapshot

To create a snapshot of the Volume you simply right-click on it and click on Create Snapshot. Or you can select the Volume and click on the Create Snapshot button at the top.

Please note that I am creating a Snapshot of the running virtual machine. If you have a database on it or some other application that has a lot of transactions happening then it would be best to turn the virtual machine off (schedule some downtime) and do the Snapshot.

Give the Snapshot a meaningful description and a name. Click on “Yes, Create”.

Once done, you will be able to see your Snapshot in the Snapshots part of the Elastic Block Store area. See example below. Please note the Name, Description and Capacity (original 8 GB).

Creating a new (clone) Volume

Next, we’ll create a Volume out of the Snapshot. Simply right-click on the Snapshot and select “Create Volume from Snapshot” from the resulting menu or you can use the button at the top.

Put in the desired Size of the new Volume and make sure the Availability Zone coresponds to the AZ that the original Instance is in. Click on “Yes, Create”.

If you navigate to the Volumes section you will see the new 25 GB volume there. You’ll notice that it is not in use.

Attach new Volume to existing Instance:

And now for the fun part. We will stop the Instance, detach the original 8 GB Volume, attach the new 25 GB Volume and make sure we are able to boot. Please note the new Volume will have the data up to the point when the Snapshot was taken. If the original machine had more data put onto it since the Snapshot then that will have to be dealt with. This is beyond the scope of this tutorial.

Stop the Instance (if it is running).

Go to the EBS Volumes section, select the original Volume, right click on it and select “Detach Volume”. You can also use the Detach Volume button. When prompted, please select “Yes, Detach”.

To attach the new Volume right-click on it and select “Attach Volume”.

Make sure you select the correct Instance. In the example below I am also modifying the Device to be the same as the original one. The example is for a Linux (Ubuntu) Instance. This is important. Click on “Yes, Attach”.

The Status should change to “in-use”.

Go back to your Instance and start it up. Log back into your Linux Instance and run the following command (for ext3 file system):

    resize2fs /dev/xvda1

Please note that the device is not called sda1 but xvda1. The device was renamed by the Kernel.
After it is done you should be able to see the full 25 GB. use the “du -h” command. Example:

Cleanup:

Once you’ve made sure all works as it should and that all your data is fine on the new Volume, you can remove the Snapshot and the old Volume if you do not intend to use them anymore. Go to the relevant AWS Management Console sections and remove (delete) them.

Connecting to AWS Linux (Ubuntu) Instance from Windows using PuTTY and the SSH protocol

By Semir H.

This part continues from the previous “NAME” part where we launched an Instance. In this part I will show you how to use PuTTY to connect to your running Linux (Ubuntu) Instance. In order for PuTTY to access your Instance over the SSH protocol we opened up the SSH TCP port (22) by using a Security Group which is essentially a Firewall.

Creating a PuTTY Private Key:

We must first create a PuTTY Private Key using the .PEM key generated beforehand. For this you need to get puttygen.
Run puttygen to start the process. Click on “File” and on “Load private key”.

Browse to the private key file (.pem) you saved earlier and load it. You should get a success notice like the one below. Click on OK.

You may want to edit the key comment and put a key passphrase before we generate a PPK file. I recommend using a key passhprase but please do not forget it or your resulting PPK will be of no use. Leave the type of key as SSH-2 RSA and leave the number of bits as 1024. Once you’ve made desired comments and put in the phrase click on “Save private key”. Give it a meaningful name and save as .ppk.

Connecting to AWS Linux Instances:

We must first find out the public DNS of the Instance we wish to connect to. To do this go to your AWS Management Console, go to Amazon EC2 tab, Instances, click on your Instance and look down under “Description”. You should see an address, next to Public DNS, similar to this: ec2-122-248-203-235.ap-southeast-1.compute.amazonaws.com

Next, start PuTTY, put the DNS entry into the Host Name field and then browse down to the SSH Category and to the Auth section under SSH. This is where you add your private key (PPK) for later authentication. Go back to the Session category, make sure the DNS entry is still there, give this session a name (e.g. Test AWS Ubunt) and click on Save.
See below for an example:

Now it’s a simple matter of loading the saved session. First time you connect you will get a security alert saying that the server’s host key is unknown to you. Please select Yes to continue if you think it is safe to do so (in this case it is). You will get this with any new server the first time. You will also get this if you turn your Instance off and then on again (because the DNS and server hostname will change). The private key will not change so that can be re-used to connect.
You will be prompted to put in a username to log in. For this particular AMI it is ‘ubuntu’. Once you put that in you will get prompted to put the PPK passphrase (if you set one). If all goes well you should be logged into your Ubuntu Linux Amazon AWS Instance. Yay :)

Troubleshooting:

If you’re having issues then please revisit the steps above. Please make sure you’re using the correct DNS entry and the correct private key. You can check which private key the Instance is using by looking at the Instance Details section (same one where DNS info is located). Please be aware that the public DNS string will change if you shut the server down and start it again. This can be addressed using Elastic IPs but we’ll cover that in another tutorial some other time.

Securing Websites with Pound (reverse HTTP proxy & SSL wrapper)

This is a short article on how to use Pound as a Reverse SSL Proxy.

Problem/Requirement:

You have a few websites in your DMZ or even on the LAN. You want them to be accessible from to the outside (Internet). Some of these websites are web based applications. You also have 1 external IP.
You want to serve all these websites from the one external IP and you want most of them to be secured via SSL (HTTPS).
On top of that you have different web servers serving the pages (e.g. Apache, IIS).

The diagram below depicts a typical inefficient web server setup (click on image to see larger version):

Requirements Summary:

• Use just one external IP.
• Use one SSL certificate (for websites under the same domain/sub domains).
  
• Secure websites REGARDLESS of the web server technology used to serve those websites.
• Have the ability to pick which websites get SSL encryption based on the source of the request (e.g. if from Internet then use SSL, if from corporate LAN then do not encrypt).

Solution:

Use a free application called Pound.

What is Pound?

Pound is a reverse HTTP proxy, load balancer, and SSL wrapper. It proxies client HTTPS requests to HTTP backend servers, distributes the requests among several servers while keeping sessions, supports HTTP/1.1 requests even if the backend server(s) are HTTP/1.0, and sanitizes requests.
It runs on Linux.
You can find more info @ http://www.apsis.ch/pound/

How Will it Work?

The diagram below shows where we want to be (click on image to see larger version):


Basically we want the Pound server to be the first point of contact from the outside. This way we reduce the number of "holes" we have to open in the firewall(s) and put another layer of security between the Internet and our web servers.
Pound meets all of our previously stated requirements:

• We can use just one External IP to server all our websites.
  
• Using a wildcard certificate (e.g. *.testdomain.test) we can encrypt all the web sites under the same domain/sub domains with just that one cert.
  
• Because we are doing the SSL encryption with Pound, it no longer matters what the other web servers inside your network use to serve pages. They can be Apache, IIS, Tomcat, etc. You can leave them ALL at the default port 80 (or 8080 for some). Pound can talk to them on pretty much any available port (even 443).
• We can make pound encrypt all requests that come from the Internet but NOT do so if the requests come from the LAN. Some companies choose to do that because it makes things faster. Not using SSL/HTTPS internaly can be done in two ways. One way is through Pound (so all the internal websites would point to the same internal Pound IP) and the other way is to not involve Pound at all. People wold just directly access the individual web servers. An advantage of going via Pound is that you can track access from one location and perhaps do custom redirects when needed (e.g. if a web server is down for maintenance you can have pound redirect to a notification page somewhere else).

Of course, there is one problem with using Pound for evertynig. It becomes a single point of failure. I suppose that can be remedied by using another Pound server to load balance or in another way (e.g. run as Virtual Machine - which works good. Let VMware worry about high availability, etc)

Next we will examine how some of the above is actually implemented:

Please Note:
This tutorial will not examine how the SSL certificates are acquired. The only thing you have to make sure is that they are Linux compatible. Pound runs on Linux. I usually use a self signed cert but if you generate one via a trusted CA then use the Apache/Linux instructions.

Let's assume you want to Encrypt web access for two of your internal web servers IF the requests are from the Internet. If they are from the LAN then do not encrypt.

Domain = testdomain.test
Internal Web Srv 1 = 192.168.7.5 (DMZ)
Internal Web Srv 2 = 192.168.0.10 (LAN)

The Pound server should always be put in the DMZ. Ideally all of your "exposed" servers should be but that is sometimes not an option.

Pound Server = 192.168.7.50 (DMZ)

First we define a HTTP (port 80) listening directive. Under that we put in our two web servers and since we want to encrypt them (if the requests is from the Internet and is on port 80 (HTTP) we will redirect to HTTPS:

ListenHTTP
Address 192.168.7.50
Port    80
xHTTP   2
Service
HeadRequire "Host: .*web1.testdomain.test.*" 
Redirect "https://web1.testdomain.test"
End
Service
HeadRequire "Host: .*web2.testdomain.test.*"
Redirect "https://web2.testdomain.test"
End
End

Next we define a HTTPS directive. This will be the one that will accept the redirects from above AND direct HTTPS requests (no redirect required in that case). This is also where we define which certificate to use and which back servers to contact.

ListenHTTPS
Address 192.168.7.50
Port    443
 Cert    "/path/to/cert.pem"
AddHeader "X-Forwarded-Proto: https"
HeadRemove "X-Forwarded-Proto"
Ciphers "ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL"
xHTTP   1

Service
HeadRequire "Host: .*web1.testdomain.test.*"
BackEnd
Address 192.168.7.5
Port 80
End
End

Service
HeadRequire "Host: .*web2.testdomain.test.*"
BackEnd
Address 192.168.0.10
Port 80
End
End
End

There is also a recommended "catch-all" deirective that goes to the end. This will be what Pound will serve if nothing above is matched. Example:

Service
BackEnd
Address 192.168.0.10
Port    80
End
Session
Type    BASIC
TTL     300
End
End

Access from the Internet:

Your external DNS will point web1 and web2 URLs to the same external IP. Your firewall (I use shorewall) will redirect traffic that comes on that IP over port 80 and 443 to the Pound server in the DMZ. Your internal DNS will point the above URLs to the DMZ IP of the Pound server (192.168.7.50).
So when the requests come from outside on HTTP they first hit the HTTP directive which just redirect to HTTPS. Internal DNS kicks in and the traffic is redirected back to Pound but this time over HTTPS (443). This is when the HTTPS directive kicks in. If you asked for web1 then request is directed internally to 192.168.7.5 over port 80 (HTTP). The answer is returned to the Pound server over port 80 (not encrypted) and then the traffic is encrypted to goioutside (from Pound to the Internet).
If the request from the outside is over HTTPS then the HTTPS directive is used straight away. Answers are returned to the Internet over HTTPS of course.

Access from the Inside:

The easy way is NOT to use pound if you don't want encryption.

If you want internal encryption then you just have to point the internal DNS to the Pound IP.

If you do not want encryption but still want to go through Pound then you need to give the Pound another internal IP (either another NIC or add an IP to the current NIC) and then create HTTP directives that forward traffic to the individual web servers. Of course, your Internal DNS will then have to point to that new Pound IP.
What happens if people have bookmarks that have HTTPS in them?
You can create HTTPS directives that redirect to HTTP. Effectivelly you do the opposite of what you did for the Internet traffic. Slightly more work but not the end of the world :)

Pound is capable of more complex setups than this and I may describe some of those in the future.