This page is offered as a service of Bristle Software, Inc. New tips are sent to an associated mailing list when they are posted here. Please send comments, corrections, any tips you'd like to contribute, or requests to be added to the mailing list, to tips@bristle.com.
Original Version: 9/29/2009
Last Updated: 10/28/2009
"Cloud Computing" is a hot topic these days. It's the "next big thing". However, it is not really a new thing, so much as a new name for a useful collection of old things, with some new twists thrown in.
Previous names for the various parts include (from the 1970's) "mainframe", "time-sharing", "virtual machine", and (more recently) "network", "LAN", "WAN", "Internet", "World-Wide Web", "hosting service", "groupware", "virtualization", "hypervisor", etc. These have all been ways for a single large computer to act like multiple smaller computers, or for multiple computers to interact with each other. Cloud Computing pulls these various concepts together in a useful way, and with a jazzy new name.
So, why now? What's driving the convergence? In a word: "broadband". That is, the recent dramatic increase in the speed and availability, and the dramatic decrease in cost, of high-speed networks. Suddenly in recent years, Joe Average can cheaply get a fast connection (DSL, cable modem, FIOS, etc.) from his home or small business to computers throughout the world. Speeds of 10, 20, or even 50 Mbps, are now common (and getting faster every day). Large corporations have had such connections for years, but they've been expensive. Those costs are now dropping fast. For the first time in history, it is typically as fast to access a computer thousands of miles away as to access the hard drive on your local computer. Or if not quite as fast, certainly "fast enough" to open a whole new range of possibilities.
Cloud computing comes in many different forms, with more emerging every day. Details in subsequent tips.
--Fred
Original Version: 10/28/2009
Last Updated: 10/28/2009
The computing "cloud" can have different access modes, and different physical locations.
Public
The first popular option was the "Public" cloud.
It typically resides outside of your building, somewhere on
the Internet. It achieves economies of scale by using a few
powerful servers to support the needs of many customers. It
is accessible to the general public, subject only to login restrictions. Anyone
with a valid username and password, or other security device, can
access the cloud. This
is very convenient, but not necessarily secure. You
have to worry about things like password-cracking, line sniffing,
and other security explots.
Private
For better security, the "Private" cloud became an option. Some
large companies
wanted all the advantages of a cloud (flexibility, scalability,
reduced electric bills, reduced physical space, etc.), but couldn't
afford the security risks. They created
their own private clouds, using the exact same technology as a public
cloud, but residing entirely in their own building, behind their own
firewall, on their own physical servers.
Smaller companies wanted to do the same thing, but couldn't afford it. Therefore a market emerged for cloud vendors to offer a private cloud option. The cloud would physically reside outside of your building, somewhere on the Internet, but would be accessible only via a dedicated line, or only via encrypted communications (SSL, VPN, etc.), and perhaps only from a restricted set of client computers, IP addresses, etc.
Hybrid
Recently, the "Hybrid" cloud has emerged as a third option. With
separate public and private clouds, communication can be
inefficient, with all traffic being routed from one cloud into your
building, through your firewall, and back out to the other cloud. With
public and private parts of a single cloud residing on separate servers
at the same cloud vendor, communications can take a secure shortcut.
--Fred
Original Version: 9/29/2009
Last Updated: 9/30/2009
SaaS (Software as a Service) includes familiar things like Web e-mail where you don't have to install the software on your own computer, or even keep the data there. You just connect to the Yahoo, Gmail or other Web site to read your e-mail, send messages, etc. Also, things like Google Docs and Google Apps where you don't have to install a word processor, spreadsheet program, or other app, on your own computer. You just connect to the Google Web site to view and edit your documents and spreadsheets. Additional such services are springing up right and left.
It is becoming possible to take a step back towards the truly "thin client" -- a computer with a display, keyboard, mouse, fast CPU, lots of RAM, but very little hard drive disk space. Almost like the "dumb terminals" of the 70's, but with more processing done locally. Having no applications stored locally means you don't have to install, upgrade, configure, patch, re-patch, re-install, re-configure, re-re-patch, all the time. Instead, you just pay a company to provide you with the latest working software at all times, and you go back to doing the things you wanted to be doing, without having to also be your own computer administrator.
Furthermore, if even your own documents, spreadsheets, e-mail messages, address book, and other data, are stored remotely, not on your own local computer, you don't have to worry about losing them. No more need to do backups. If your hard drive crashes, or your house burns down, or your computer is stolen, or you decide to buy a new or additional computer, there's no setup time. Just power on a new computer, fire up a Web browser to connect to your remote data and applications, and get started. If you move everything to a remote server "in the cloud", you may be able to get rid of your hard drive entirely, and just boot the computer from a CD or other read-only media. In that case, there's no need for a virus scanner, since there's nothing to infect with a virus. Simply re-boot the computer and it's guaranteed to be virus free.
Since your files are not on the local computer in your home or office, you can access them from any computer in the world. They are just as available to you at home as at work, or at a friend's house, or an Internet cafe, or even your cell phone (which is also a computer, after all). Others can also access your files, in controlled ways that you choose to allow. You can allow your friends and colleagues to see your pictures, your calendar, etc. You can allow them to directly edit documents on which you are collaborating, without having to send files back and forth and keep track of who is currently editing each file.
This form of Cloud Computing offers the highest level of convenience, service, support, and automation, but the lowest degree of flexibility, portability and control. What it allows you to do (send e-mail, manage docs, run apps, etc.) is very easy, but you have to do things the way the site was designed for you to do them. Furthermore, you may get locked into that one site, and not be easily able switch to another vendor. So far, most vendors seem to do a pretty good job of supporting import and export, but that's worth checking before you commit too fully to one vendor.
SaaS is ideal for computer users who don't have the skills and/or interest in computer programming and computer administration.
--Fred
Original Version: 9/29/2009
Last Updated: 9/30/2009
PaaS (Platform as a Service) includes things like Google App Engine where you, as a computer programmer but not necessarily a computer administrator, can develop entire applications remotely.
It offers a lower level of convenience, service, support, and automation, than SaaS, but additional flexibility and control (though perhaps not additional portability). It doesn't provide you with a suite of useful, general-purpose apps, but rather with a set of tools that are only useful if you are a computer programmer, planning to write your own apps. In that case, it offers many of the same advantages as SaaS, relieving you of all the administrative chores. There's no need to install/upgrade/configure the programs that make up the development environment -- compilers, debuggers, editors, etc., and no need to backup your files in case of disk crash or other disaster. As with SaaS, just power on a new computer, fire up a Web browser to connect to your remote development environment and begin programming.
As with SaaS, your program source files are not on the local computer in your home or office, so you can access them from any computer in the world. You can work from anywhere. Also, you can easily collaborate with other programmers, sharing access to the same set of source files.
As with SaaS, however, you have to do things the way the site was designed for you to do them. You may get locked into that one development environment, and not be easily able switch to another technology or even another vendor supporting the same technology. Hopefully, the vendors offering these services will be responsive to the needs of their users. For example, at first Google App Engine supported only Python, but it recently added support for the Java JVM, which supports Java, Groovy, JRuby, Scala, Clojure, AspectJ, and others. It does not yet support the use of a standard SQL relational database, like MySQL. Instead, you have to use Google's BigTable, GQL, etc.
PaaS is best for computer programmers who don't have the skills and/or interest in computer administration, and who are willing to limit themselves to the technologies offered by the PaaS vendor.
--Fred
Original Version: 9/29/2009
Last Updated: 9/30/2009
IaaS (Infrastructure as a Service) includes things like Amazon Web Services (AWS) where you, as a computer adminstrator, can set up a "virtual" standard Linux or Windows server.
You log in as an admin or "root" user and install/configure any applications and tools you choose. It is exactly like setting up a physical server by buying an actual computer, plugging it into your own electrical outlet and your own Internet connection or LAN, logging in, and installing/configuring the apps and tools. The big difference is that your "virtual" server is actually a simulation running, along with potentially many other such simulations, in a large server somewhere else, with the IaaS vendor buying the hardware, paying for the electricity and the high-speed Internet connection, etc. Also, it is much more efficient than running your own physical server, because the single larger server sits idle much less than your own small server might have, and consumes less electricity and less air-conditioning than all of the small servers would have, etc.
This is accomplished by minor tweaks to mature technology that has been around since the 1970's. A "hypervisor" program runs on the big server, managing multiple concurrently running instances of the same or different operating systems. This is much like the way an operating system manages multiple concurrently running programs, and the way IBM mainframes have always run multiple "virtual machines", one for each logged in user, within the mainframe. Each virtual server appears to have its own disks, its own RAM, its own CPU, etc., but is really sharing larger disks, more RAM, and a faster CPU with other virtual servers within the hypervisor.
IaaS offers a lower level of convenience, service, support, and automation, than SaaS or PaaS, but much more flexibility and control, as well as absolute portability. It doesn't necessarily provide you with a complete suite of useful, general-purpose apps like word processors, spreadsheets, calendars, etc. Nor does it necessarily provide you with a complete programming environment. Instead, it provides you with a server that may be preconfigured with a standard Linux, Windows, or other operating system, and the standard collection of applications and tools that come with that operating system. You may also choose a server that comes preconfigured with additional standard portable tools and applications. And you can install or write as many other non-standard tools and apps as you like.
IaaS does offer some of the administrative advantages of SaaS and PaaS. For example, there may be no need to backup your files in case of disk crash or other disaster. The vendors typically offer backup or data-duplication services that protect you from any single point of failure. So, you should never lose anything at the server, and you don't have to store anything on the local client computer. As with SaaS and PaaS, just power on any client computer, connect to the remote server and begin working.
However, it typically does not offer the SaaS and PaaS advantage of updating and patching the operating system, applications, and tools. You choose an initial server configuration, and are responsible from then on for any updates and patches that may be required. More control (you never get a patch that adds a new bug, just at the time you can least afford it), but more work for you to do yourself. On the other hand, just like with a physical server, you can choose to use an "automatic updates" type of service to apply updates, patches etc. for you. And you can hire a service to perform any administrative tasks you don't want to bother with.
As with SaaS and PaaS, your server does not physically reside in your home or office, so you can access it from any computer in the world. You can work from anywhere. Also, you can easily collaborate with other users and programmers, sharing access to the same set of apps, tools, and files.
The real advantage of IaaS is that you don't have to do things the way the vendor intended. You have complete control over your own server, and can do whatever you want (subject to rules against spamming, illegal activities, etc.). You can't get locked into one development environment, one technology or one vendor. If you don't like the first IaaS vendor you choose, simply set up a server at another vendor (or go back to your own hardware), and copy all of your portable operating system configurations, apps, tools, and files there.
This is especially convenient if you are using open source apps and tools written in languages like Java and running on operating systems like Linux, where no "install" is typically needed. In that case, you simply copy all the files from one server to another and resume your work. No need to deal with issues of transferring software licenses from one computer to another, or running install programs to re-install the software on the new computer, or manually repeating hours or days of pointing and clicking to re-configure the operating system and applications, and then wondering what you missed that is causing it to behave differently. Just one single rsync command to copy all the files, and you are on your way.
IaaS is very simple -- there is nothing new to learn, other than how to launch and terminate server instances. If you already know how to configure and administer a Linux or Windows server, your skills are completely portable. You still do everything the way you always did. Instead of just having a Web mail interface or doc/spreadsheet interface (like SaaS), and instead of having to develop software to fit a specific proprietary framework (like PaaS), you have complete control of the server in a portable way. Once the server instance is launched, even if you don't have a fully configured server somewhere else to copy from via rsync as described above, you log in as usual, create/delete user accounts as usual, install Tomcat, MySQL, etc. as usual, deploy your apps and other files as usual, etc.
IaaS is ideal for computer users or programmers who have the skills and interest to do their own computer administration, and who value such control, flexibility and portability.
--Fred
Original Version: 9/29/2009
Last Updated: 9/30/2009
For more info, see:
http://www.webguild.org/2008/07/cloud-computing-basics.php
- Good definitions of SaaS, PaaS and IaaS
http://www.webguild.org/category/cloudnomics
- Aggregated newsfeed of cloud computing articles from various sources
http://cloudfeed.net/2008/06/03/defining-saas-paas-iaas-etc/
- Very well-written daily blog on cloud computing
--Fred
Original Version: 8/12/2009
Last Updated: 10/29/2009
Amazon Web Services (AWS) is a paid service that allows you to
create virtual Linux and Windows servers to replace your physical
servers. It is the type of "Cloud Computing" known
as IaaS (InfraStructure as a Service).
There are several different services, so you can pick and choose among
them:
Inexpensive:
It can be very inexpensive to replace your physical servers with
virtual servers. It now costs me about $62/month (8.5 cents/hour)
for a virtual server that has a faster CPU, more RAM, more disk space,
and a
much faster Internet
connection, than the physical server I replaced. If I commit
to a one-year or three-year subscription, I can drop that price to $41/month
or even $32/month. I was previously paying $120/month for just
the DSL line to the physical server, since I needed a fixed IP address
and a fast upload speed. I also had the costs of originally buying
the server, occasionally buying new hardware (like a new UPS every couple
years), paying for electricity, etc. When it came time to do a
major upgrade to a newer, bigger/better/faster server, I moved to
Amazon instead. It is now blazingly fast, especially the upload/download
speeds.
Convenient:
It can also be very convenient to have a virtual server instead of a
physical server. I no longer have to worry about electric outages, DSL
phone line outages, being there in person to reset modems and
routers, etc. That's all handled by Amazon. My virtual server is
really running on one of their powerful physical servers, and they
already have redundant power supplies, redundant comm lines, etc. My
uptime is greatly improved, and I don't have to do any of the work.
New possibilities:
With virtual servers, new possibilities arise. Unlike physical
servers, it is very easy to create, delete, and clone them. No
more waiting for a physical server to be purchased and shipped to you. It
takes about 60 seconds to create an additional server that is a clone
of an existing server. You can easily create 100 more servers to
handle the busy shopping season, if you are an e-commerce site. You
can easily stop your servers over the weekend to save money when they
are not needed, since you only pay for the hours that they are running.
Once, when I was installing software on a server, I noticed a
particular file and wondered whether it had always been there, or had
been created by the install. To find out, I quickly created
another server, looked for the file there, and deleted the server. Since
the additional server was up and running for less than an hour, this
experiment cost me less than 10 cents.
--Fred
Original Version: 8/12/2009
Last Updated: 2/11/2010
To use Amazon AWS, you must create an AWS account and sign up
for each of the services you want to use.
Go to:
http://aws.amazon.com
and sign up for a free AWS account, giving your e-mail address, a
password, contact info, etc. Read the customer agreement. All of
your data and apps belong to you, not to Amazon. You're not
allowed to use it send spam, attack other computers, or do anything
illegal.
Then, go to:
http://aws.amazon.com/s3
and enter a credit card number to sign up for the S3 service that
allows you to store data at Amazon. You can do this, even if you
don't use the EC2 service and thus don't have a virtual
server. You can use the storage space to copy files to/from your
desktop or any physical servers you may have. Or you can skip this
step, if you want to run a server, but have no need to save
persistent files because, for example, all of the Web pages and
Java code that runs at the server can easily be re-published to
there from your desktop.
Then, go to:
http://aws.amazon.com/ec2
and authorize the use of the credit card to sign up for the
EC2 service that allows you to create virtual servers at Amazon.
So far, this is all free. You have given Amazon a credit card
number, but there is no charge except for hours that your virtual
servers are running, and the gigabytes of storage you are using --
none yet, in both cases.
For more info, see:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?using-credentials.html
--Fred
Original Version: 8/14/2009
Last Updated: 2/11/2010
Before you create any Amazon EC2 instances, you'll want to set
up security for them.
Note: The AWS console has recently been enhanced to allow you
to set up security on the fly, while creating your first instance,
but that wasn't an option when I did it, so I did these explicit
steps.
X.509 Certificate and AWS Access
Key Identifier (Optional):
While creating your AWS accounts, you can optionally take the
further steps of creating an "X.509 Certificate", and an "AWS
Access Key Identifier". However, these are needed only if you
intend to access the AWS services through the SOAP or REST web
service interface, or perhaps by the CLI (command line interface)
that uses the SOAP interface behind the scenes. If you plan to
always use the AWS Console web page to manage your virtual servers
and your virtual hard drives, you don't need to bother. I created
them, but probably didn't really need to, since Amazon keeps adding
more capabilities to the Console, and I can now do everything
there. For more info on creating them, see:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?using-credentials.html
Firewall:
In all cases, you must configure the EC2 server's firewall. It
defaults to having no ports open, so you can't login to your server
instances, people can't see your Web sites, etc. Here's what
I did:
http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/StartConsole.html
http://clouddb.info/2009/05/17/using-and-managing-aws-part-3-aws-security/
--Fred
Original Version: 8/14/2009
Last Updated: 1/15/2010
--Fred
Original Version: 8/14/2009
Last Updated: 11/7/2009
--Fred
Original Version: 8/14/2009
Last Updated: 11/12/2009
While logged in to the server as root, set the hostname of the
server to whatever you want, instead of having to use the default
value assigned by Amazon, which is based on the internal IP address
used within the Amazon LAN. For example, to set a Linux Fedora
server to be known as "myhost" with a fully qualified domain name
of "myhost.mydomain.com":
--Fred
Original Version: 8/15/2009
Last Updated: 9/7/2009
Problem: Dynamic IP
Address
When you launch a server instance, it is dynamically assigned an IP
address by Amazon. That IP address will stay assigned to your
server as long as your server is running, even if you re-boot the
server. However, if you "terminate" the instance (via the AWS
Console, for example), so that you stopped paying for it per hour,
the IP address is released to the pool of IP addresses available
for use by other Amazon customers, and may soon be assigned to a
different virtual server. Later, when you launch a new server
instance, the new server is assigned a different IP address, even
if you launch the new instance from an exact AMI image of the old
server -- one that you created by "bundling" the server into an AMI
and uploading it to the Amazon S3 service.
Solution: Elastic IP
Address
To preserve the same IP address, allocate an "elastic" IP address
from Amazon. You can allocate one or more such addresses, and can
assign one to each of your servers. When they are not assigned to
any server, they cost you one cent ($0.01) per hour. When they are
assigned to a server, they are free, included automatically in the
hourly cost of the server. Elastic IP addresses are addresses that
you have reserved for as long as you like. They will not change if
you terminate and create server instances. You can move them back
and forth between servers quickly (seconds), so they take effect
much more quickly than DNS server changes, which can take hours to
propagate to all of the worldwide DNS servers.
To allocate an elastic IP address:
The elastic IP address is immediately (within seconds) assigned,
and can be used by your users world-wide to access your server.
Since it is now assigned to a server, you are no longer billed one
cent per hour for it. The IP address previously assigned to the
server is released to the pool of IP addresses available for use by
other Amazon customers, unless it was another elastic IP address of
yours, in which case it becomes an unassigned elastic IP address
for which you are billed one cent per hour, and which you can
immediately assign to a different server if you choose.
Each Amazon IP address (elastic or dynamic) has an associated
public DNS name. For example, the IP address 100.101.102.103 might
have the public DNS name
"ec2-100-101-102-103.compute-1.amazonaws.com". Therefore, when you
change the IP address of your server, the public DNS name changes
also. To avoid this, you may want to assign your own DNS name, as
described in the next tip.
For more info, see:
--Fred
Original Version: 8/16/2009
Last Updated: 8/16/2009
Now that you have a stable IP address (an Amazon AWS "elastic" IP
address), you can map any DNS name you own to that IP address. For
example, I configured the bristle.com DNS server to map the name
bristle.com to my IP address. Thus, whenever
anyone in the world refers to the name
bristle.com, they are directed to my server.
Unfortunately, there is no way to get Amazon's DNS server to stop
mapping its generated DNS name to the same IP address.
Therefore, whenever anyone in the world refers to that generated
name (for example,
ec2-100-101-102-103.compute-1.amazon), they are
also directed to my server. That's okay. There's no real problem
with multiple names mapping to the same IP address.
A bigger problem is the fact that there is no way to get Amazon's
DNS server to update its reverse DNS mapping.
That is, it still maps my elastic IP address to the generated DNS
name ec2-100-101-102-103.compute-1.amazon, not to
my preferred name bristle.com. Furthermore, since
the Amazon DNS server, not the bristle.com DNS server, is
responsible for the reverse DNS mapping of all IP addresses owned
by Amazon, I can't do the desired reverse DNS mapping in my
bristle.com DNS server. As the current worldwide DNS system works,
there can only be one reverse DNS mapping for a given IP address.
Therefore, whenever anyone in the world looks up the IP address of
bristle.com, they correctly get my elastic IP
address, but if they then check the validity of that mapping by
doing the reverse DNS lookup of that IP address, they get the
Amazon generated DNS name
ec2-100-101-102-103.compute-1.amazon instead of
the expected bristle.com.
This is a problem especially if I run an SMTP (e-mail) server on my
server. When I hand off e-mail to other SMTP servers that want to
prevent spam, they may check my reverse DNS lookup, and falsely
label me a spammer. I haven't yet found a solution to this, other
than configuring my SMTP server to relay through another
cooperating non-Amazon SMTP server. Any better ideas?
--Fred
Original Version: 9/4/2009
Last Updated: 11/12/2009
You'll want to have an SMTP server running. This allows your
users to send e-mail. Perhaps more importantly, it allows
automatic processes like cron jobs, tripwire, logwatch, etc., to
send e-mail. You'll sometimes want that e-mail to go to other
computers, not to local mailboxes. For example, you may want
messages about security to go to your primary e-mail account.
The biggest problem you'll encounter is that other computers may
reject your e-mail as spam, for reasons given in Point the DNS at the IP Address.
My best solution, so far, is to relay all outgoing mail through
another SMTP server -- one that is not suspected of being a spam
source. You should be able to use any SMTP server that you have
legitimate access to: your ISP, Google Gmail, Yahoo Mail, or
whatever.
However, be warned that such servers may impose limits on how much
mail they will allow you to send. You'll be configuring your SMTP
server to act like an MUA (mail user agent), not an MTA (mail
transfer agent), so it will look to the remote SMTP server as
though you are personally sending all of the mail. It may cap you
at some number of messages or bytes, per day or month. Beyond
that, it may reject, delay, or even discard messages. The company
may assume your computer is infected with a virus that is sending
all the mail, or that you are a spammer, or may simply say you are
violating their rules. Furthermore, they may not allow you to send
mail from the multiple usernames on your server, or from usernames
at a domain name different from theirs. Be sure you know their
policies before relying on them to relay your mail. If necessary,
pay an "SMTP Mail Relay Service" for the right. See: http://google.com/search?q=smtp+mail+relay+services.
I use a service that I was already paying to host the incoming
e-mail for bristle.com.
Here's what I did with my SMTP server (sendmail):
For more info, see:
http://pauldowman.com/2008/02/17/smtp-mail-from-ec2-web-server-setup/
http://clouddevelopertips.blogspot.com/2009/06/sending-email-from-ec2.html
http://blog.twinklesprings.com/2008/03/27/remote-mail-delivery-for-google-apps-and-postfix-mail-server/
http://www.birds-eye.net/article_archive/smtp_mail_relay_services.htm
http://dbaron.org/linux/sendmail
http://www.madboa.com/geek/sendmail-genericstable/
http://does-not-exist.org/roessler/genericstable.html
http://www.sendmail.org/m4/features.html
http://www.brandonhutchinson.com/Sendmail_masquerading.html
http://www.howtoforge.com/configuring-sendmail-to-act-as-a-smarthost-and-to-rewrite-from-address
--Fred
Original Version: 8/20/2009
Last Updated: 2/11/2010
Now you have a usable server that can be found by anyone in the
world based on its name. That may be as far as you need to go. If
the server doesn't need to have any persistent data, you may be
done.
For example, I have one Web site that shows static Web pages, as
well as pages that are generated programmatically by Java code
running in a Tomcat Web server. However, users of that Web site
cannot enter any data to be saved at the site. They can browse the
site, but not fill out forms, make purchases, enter data into a
database, etc. That site has no need to maintain persistent data.
If I were ever to terminate and re-launch the instance, or if the
real Amazon server hosting my virtual server were to crash, I would
lose all of the local data, but that's not a problem because I can
recreate it all by re-publishing all of my static Web pages, Java
code, etc.
However, most servers need persistent data. You may want
configuration changes that you make after launching the instance to
be preserved, even if you terminate and re-launch the instance, and
even if the real Amazon server crashes while hosting your virtual
server. You may also want to preserve log files from your Apache
Web server, Tomcat server, database server, etc. And to preserve
local databases where your Web applications may store data entered
by your users. The solution is to use not only the Amazon EC2
service to run a virtual server, but also the Amazon S3 service to
store files, and the Amazon EBS service to make those files
accessible to your virtual server. Since EBS volumes are
automatically replicated across multiple Amazon disk drives on
multiple Amazon servers at different physical locations, a
single disk crash or other local disaster won't lose your data.
Here's what I did:
Create an EBS volume and attach it to your server
instance:
Note: Do NOT do this step if you are mounting an EBS volume that already contains a file system and files that you want to preserve. It deletes all files from the EBS volume.On my Linux server, I did:
I did some tests, and found that my virtual server can read/write
files on the EBS volume as fast or faster than on its own file
system, so performance doesn't seem to be an issue.
For more info, see:
http://aws.amazon.com/ebs/
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-ebs.html
--Fred
Original Version: 8/24/2009
Last Updated: 11/25/2009
Your server is now ready to use. The next step is to configure it to do what you want it to do. In my case, I created additional usernames, copied a bunch of my files there, and installed and configured various servers: Apache HTTP server, Tomcat server, MySQL, etc.
Much of the configuration may already be done. As I mentioned in Launch an EC2 Server Instance, when choosing an AMI to use as the template for your server's initial configuration, you can choose one that is pre-configured with various combinations of useful software packages. For example, the "Java Web Starter" AMI already has Java, Tomcat, MySQL, and Apache HTTP Server installed and configured. In fact, I started with that AMI, but later went back and started over so I'd have full control over the configuration and full knowledge of how to re-create it from scratch, if necessary some day. I've also noticed that the "Getting Started on Fedora Core 8" AMI already has Apache HTTP server installed, unlike the "Basic Fedora Core 8" AMI that I used.
Here's what I did:
--Fred
Original Version: 8/25/2009
Last Updated: 8/25/2009
Now that you've installed a bunch of apps, you may want to move
their data to persistent storage, so you won't lose it if you
terminate and re-launch the instance, or if an Amazon server
crashes while hosting your virtual server. You don't have to
re-install or re-configure the individual apps -- you can simply
use symbolic links to redirect them to the EBS file system. Here's
what I did:
Move tree of Web pages served by Apache HTTP
server:
% mkdir -p -v /ebs/var/www/html
% mv /var/www/html/* /ebs/var/www/html
% rmdir /var/www/html
% ln -s /ebs/var/www/html /var/www/html
Move Tomcat log files:
% /etc/rc.d/init.d/tomcat5 stop
% mkdir -p -v /ebs/var/log
% mv /var/log/tomcat5 /ebs/var/log/tomcat5
% ln -s /ebs/var/log/tomcat5 /var/log/tomcat5
% /etc/rc.d/init.d/tomcat5 start
Move MySQL database files and logs:
% /etc/rc.d/init.d/mysqld stop
% mkdir -p -v /ebs/var/lib
% mv /var/lib/mysql /ebs/var/lib/mysql
% ln -s /ebs/var/lib/mysql /var/lib/mysql
% mkdir -p -v /ebs/var/log
% mv /var/log/mysqld.log /ebs/var/log/mysqld.log
% ln -s /ebs/var/log/mysqld.log /var/log/mysqld.log
% /etc/rc.d/init.d/mysqld start
Original Version: 8/26/2009
Last Updated: 9/18/2009
Once you've gained enough confidence in the whole virtual server
idea, you may want to save some money by "reserving an instance".
Basically, that means that you pay up-front for a 1- or 3-year
subscription, and your cost goes way down. Here's the relative
prices:
| Subscription length | Prepaid cost | Hourly cost | Total Cost (to run 24x7) |
| 0 years ("On-Demand") | $0 | $0.085/hour | $62/month |
| 1 year | $227/year | $0.03/hour | $41/month |
| 3 years | $350/3 years | $0.03/hour | $32/month |
Whenever you have an instance running, if that instance matches the
parameters (size: small, medium, large, etc., operating system
type: Linux, etc.) of an instance you have reserved, you get the
lower reserved instance price of $0.03/hour instead of the higher
"on-demand" price of $0.085/hour. For example, if you reserve
an instance, then launch 2 instances that match it, the 1st one gets
the lower reserved price, and the 2nd one gets the higher on-demand
price. If you then terminate the 1st instance, the 2nd one
switches automatically to the lower reserved price.
For more info, see:
http://aws.amazon.com/ec2/reserved-instances/
--Fred
Original Version: 10/15/2009
Last Updated: 10/15/2009
Amazon helps you predict the amount you will be billed for their services, and lets you see the current balance of your next monthly bill.
To predict your future bills, even if you have never signed
up for an AWS account, use the calculator at:
http://calculator.s3.amazonaws.com/calc5.html
To see the current balance (to within a few hours) of your AWS account:
You'll see a page that details your EC2 instance charges ($0.085 or $0.03 per hour, with the number of hours so far and a total cost -- always about $22/month for me), plus your EBS storage charges (always $3.00/month for me since I've reserved 30GB of EBS storage). It also shows your EC2 and EBS bandwidth charges and your S3 storage charges, but these have always been negligible for me -- 20 cents per month, or less.
--Fred
Original Version: 2/21/2010
Last Updated: 2/23/2010
Once you have your server up and running, hackers will immediately start trying to attack it. Therefore, you should set up some additional security measures, like:
For details, see Unix
Security.
Anyone have any other favorites? Also, does Windows have any active
security monitoring features like these?
--Fred
Original Version: 2/11/2010
Last Updated: 2/11/2010
Once you have a server instance configured exactly as you like it,
you may want to bundle it into an Amazon Machine Image (AMI), so that
you can create more instances exactly like it.
To do so, you
"bundle" the instance, upload it to an S3 bucket, and register
it as an AMI. After that you can launch a new
instance of the AMI at any time, and you get an exact clone of the
original instance. This is useful if you ever intend to terminate
a configured instance and want to be able to re-launch it quickly. It
is also useful as a way to clone an instance, creating multiple identical
copies.
The AWS console does not yet offer a way to bundle and upload the instance,
so I did that from the Linux command line of the instance itself. (Actually,
the "Instance Actions" menu in the "Instances" screen
of the AWS Console does offer "Bundle Instance (S3 AMI)",
but for me, it is always disabled, so either it's not quite ready yet,
or I'm doing something wrong. Any ideas?)
Gathering security info:
The Amazon CLI commands ec2-bundle-vol and ec2-upload-bundle are
pre-installed on your Linux server instance, but require local copies
of your X.509 certificate file and private key file. You may
already have generated these and stored them on your client
computer for use with the CLI or SOAP
interface. If
not, here's how:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-credentials.html#using-credentials-certificate
Once created, they have names like:
cert-32LettersAndNumbers.pem
pk-32LettersAndNumbers.pem
When you create them, you are also told your Amazon "Access
Key ID", and your Amazon "Secret Access Key", which you need
below. If you don't remember them, you can look them up:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-credentials.html#using-credentials-access-key
Finally, you need to know your
numeric Amazon "user id", which is the same as your Amazon "account
number" or
"account id", but without the hyphens. You can see it at the
top right of the "Account Activity" page, as described in:
Manage Your AWS Billing
or can look it up directly:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-credentials.html#using-credentials-account-id
Creating the AMI:
Here's what I did:
On my client computer, copying to my server instance:
- Use scp to copy my X.509
certificate and private key to my home directory (~)
On
my Linux server instance:
% sudo mv -i -v ~/cert-32LettersAndNumbers.pem /mnt
% sudo mv -i -v ~/pk-32LettersAndNumbers.pem
/mnt
% sudo ec2-bundle-vol
-d
/mnt
-k /mnt/pk-32LettersAndNumbers.pem
-c
/mnt/cert-32LettersAndNumbers.pem
-u MyNumericAmazonUserIdWithoutHyphens
-r
i386
-p MyUniqueBundleName
% sudo ec2-upload-bundle
-b MyS3BucketName
-m
/mnt/MyUniqueBundleName.manifest.xml
-a MyAmazonAccessKeyId
-s
MyAmazonSecretAccessKey
% sudo rm -i -v /mnt/pk-32LettersAndNumbers.pem
% sudo rm -i -v /mnt/cert-32LettersAndNumbers.pem
% sudo rm -i -v /mnt/MyUniqueBundleName*
% sudo rmdir -v /mnt/img-mnt
At the AWS Console:
- AMIs | Register new AMI
- AMI Manifest Path = MyS3BucketName/MyUniqueBundleName.manifest.xml
Notes:
--Fred
Original Version: 2/11/2010
Last Updated: 2/11/2010
Creating a new server instance from your own AMI is just like creating
one from any other AMI. See:
Launch an EC2 Server
Instance
The one difference I've found is that, for some reason, my own AMI's
are shown in the AWS Console without any Launch buttons. So instead
of:
AWS Console | Amazon EC2 | Launch an Instance
or:
AWS Console | Amazon EC2 | Instances | Launch Instance
I had to do:
AWS Console | Amazon EC2 | AMIs | select one | Launch
Once the new instance is created, you can use it just like
the old instance. All usernames, passwords, installed software, configurations,
etc., are identical (except that any EBS volumes are replaced with local copies
-- see below).
If you don't plan to terminate the old instance, the new instance may be a little
too identical. You
should change the host name:
Set the hostname
So far, the world doesn't know about
the new instance, so it will continue to use the old instance.
If the old instance kept files on an EBS volume, and you want the new instance
to do the same, rather than use its local copies of the files from the EBS volume,
skip the rest of this tip and go to:
Create
an Instance From Your AMI With Some Files on EBS
This is especially important if you plan to terminate the old instance,
replacing it with the new instance, and have dynamically changing files
(log files, DB files, etc.) on an EBS volume, and don't want any of
those changes to be lost during the transition.
If there was no EBS volume, you can notify the world about the new instance by
assigning it an Elastic IP address (use the same one if you
are replacing an old instance):
Set
an Elastic IP Address
and pointing your DNS server to it (already done if you used the same IP address):
Point
the DNS at the IP Address
--Fred
Original Version: 2/11/2010
Last Updated: 2/11/2010
If you have some of your files on an EBS volume, as described in:
Mount
a Persistent EBS Volume
and
Move
Files to the EBS Volume
the steps above:
Create
an AMI From Your Instance
and:
Create
an Instance From Your AMI
create an instance with a non-EBS copy of the EBS files. They
do not create a 2nd instance that accesses the same EBS volume as the
original instance.
This makes sense since Amazon has a restriction that an EBS volume can
be associated with only one instance at a time, presumably to avoid
having to deal with concurrent updates to a single file system by multiple
servers.
If you don't want to leave the files of
the new instance in non-EBS storage, you can repeat the instructions
in:
Mount
a Persistent EBS Volume
and
Move
Files to the EBS Volume
to create a new EBS volume and move the files there.
Since I planned to terminate the first instance, and wanted the
same EBS volume to be accessed by the new instance, I did this instead:
On the new instance:
% sudo mv /ebs /ebs.old
On
the old instance:
- Check for current activity, warn
users of a temporary outage, etc.
- Stop server processes that use the
EBS files so the drive can be unmounted:
% sudo /etc/init.d/mysql
stop
% sudo /etc/init.d/tomcat
stop
% sudo /etc/init.d/httpd stop
% sudo umount /ebs
At the AWS console:
- EBS | Volumes:
- Select the desired
volume
- Detach Volume
-
Attach Volume
- Instance
= the new instance in the same zone
- Device
= /dev/sdf
- Attach
On the new instance:
% sudo mkdir /ebs
% sudo mount /dev/sdf /ebs
% shutdown -r now
At the AWS console:
- Elastic IPs
- Select the elastic
IP address
- Disassociate
- Associate
- Instance
= the new instance
The new instance now replaces the old instance, accessing the same EBS volume,
using the same Elastic IP address (but a different Amazon internal IP address),
the same host name, etc. All
dynamic files on the EBS volume (log files, DB files, etc.) are absolutely current. They
were updated on the EBS volume by the old instance until its Tomcat and
MySQL servers were stopped. They were in place on the EBS volume
for further updates by Tomcat and MySQL before the Elastic IP address made it
possible for the DNS server to find the new instance. Nothing slipped
through the cracks, even if users around the world tried continuously to access
the server. The server at that IP address was cleanly down from when you
stopped Tomcat and MySQL on the old instance till you moved the IP address to
the new instance. They'll have seen a brief outage, but no unexpected behavior
and no lost data.
As soon
as you are comfortable that the old instance is no longer needed, you can terminate
it at the AWS Console.
As a sanity check, you may want to compare the
copy of /ebs made when the AMI was created with the latest copy on the EBS volume. If
they compare OK, you can delete the old copy. Here's what I did:
% sudo diff -r /ebs /ebs.old | less
% sudo rm -i -v -R /ebs.old
--Fred
©Copyright 2007-2010, Bristle Software, Inc. All rights reserved.