No matter what the service, no matter what the purpose, no matter what the product, the cloud is being touted as an answer to scalability by pretty much every major vendor. It started with data-storage, and rapidly moved to databases. In the past we were limited by the actual hardware that we owned. If we needed to expand on it, we needed to purchase more memory, and in many cases we needed to purchase new hardware as the old wore out, or was incapable of handling what we needed it to do.
However, we also had full control over the functionality; we got to know our systems and could modify them to meet our needs. Sure this was a lot of work, and required a lot of maintenance, but they functioned reasonably well, and typically the only continued cost was the human hours spent maintaining them.
Now, we are told, if we move everything to the cloud, everything will run without much effort at all. All we need to do is to simply allocate more resources and at the push of a button we will have more available to us. As a result we have migrated many of our database holdings to cloud services.
This has, however, provided mixed results. Let's break this down into the good, the bad, and the ugly, and then try and find some solutions to work it all out.
Moving to the cloud for database management does have some genuine advantages. In many cases there is a lot less need for hands-on management for most processes. Some aspects of these services really do run effectively, and many of the automated processes take away some of the grunt work associated with on-premise DB management.
Flexibility within the cloud database services is certainly better. Scaling up and down is a relatively seamless process, and it makes it considerably easy to adapt quickly to meet company needs. As a result, cloud hosted databases are excellent for testing and development. The cloud makes it easy to quickly set up a new server and determine how well it works. You also have some flexibility with platforms. For example, while not exactly the same as MS SQL Server, Azure databases are very agile. It's also relatively easy to move your databases into the cloud; servers can be set up extremely quickly and all you need to do is transfer the data; the interfaces tend to be very straight forward.
This flexibility also makes it easy to manage peak loads without a huge amount of intervention. In fact to solve most of the complicated problems you had before, you can simply move resources around or add more.
I gather you are already seeing one of the flaws here. Simply adding more resources may be effective, but it's not particularly efficient.
While with our on premise databases, we own what we have purchased, in the cloud everything is on a pay-per-usage model. Every time you add resources, there's an increased cost. This is exactly how and why the hosting providers act this way. It can become very expensive to simply throw resources at a problem and the expense can grow rapidly if left unchecked.
With your locally-hosted servers, there is considerably more functionality. Sure these tend to need a lot more maintenance, but you also have a much larger suite of tools to work from; you can get your hands dirty and solve any problems that come up yourself. One of the reasons why the cloud-based hosts are easier to manage? They have a lot less functionality. While streamlined, they are essentially limited versions of what you are used to running; much of the access you have been used to just doesn't exist.
And then it gets more complicated. Without going into too many details, throwing more resources at a problem is like sweeping it under the rug. You aren't solving the problem, you're just hiding it.
- As mentioned before, with each bit of resources being added, the expense increases. The cloud hosting providers love this, but of course your company doesn't. You still have to manage these costs. You now need to determine whether or not resources are being over- utilised. This never used to be as much of a problem, but now in the cloud the meter is constantly ticking. Getting around this can become difficult. Because of the reduced functionality in the cloud, solving real problems can become complicated quickly.
- Those offerings that supposedly made it easier to manage peak loads in the cloud? Counter- intuitively, they can result in a lot of extra work for the DBA. Maybe you had some good monitoring tools that you created that worked great on your local servers. They might have been a mess, and often sluggish, but you knew how they ran. Getting them to work with the cloud means developing a whole new set of tools. Sure the cloud services will provide you with their own monitoring tools, but they may not be anywhere as flexible as something you had for your on-premise servers.
- Another problem is that these cloud services are not quite as homogeneous as the providers would have you believe. They might be running the same platform, but they are typically running on different servers, with unknown hardware, and located who-knows-where. For these reasons, often one instance will be running completely differently than a supposedly identical one in a different location.
With your local servers, your main challenge was to make sure you had enough resources to deliver performance, or to find a way to make sure that you could squeeze the best performance out of what you have. The needs involved with managing databases in the cloud are in some ways the same as on-premise, but sometimes different. You still need to make sure that you can deliver good performance out of what you have, but not achieving good performance will also cost you financially. On top of that, if you have resources you aren't even using, and unlike on your on- premise servers, you don't own them; you are still paying for them every day. So you have to balance what you need and what you have on a constant basis, and that requires constant and vigilant monitoring.
Moving to the cloud may be helpful in some ways, but it does not reduce the amount of work required for the DBA; it only moves it around. However, one way of making sure things run more efficiently, no matter which approach you take (or are forced to take), is to have good tools which can help you monitor each database and instance, regardless of where it lives.