The total amount of data in the universe is growing by 40% a year, and we want to store and access it. A lot of it ends up in our databases. We need to scale our database instances to cater to both the growth in data volumes and the increased traffic and workload.

This strains our database engines as we push ever-increasing demands on indexes, searches, references, updates, reliability and availability.

How is database farm management different from instance management? Why do you need it and when? That is what I will try to shed some light on in this blog. If you are responsible for a database farm, read on.

Managing instances – watching and tuning performance, handling incidents, and generally maintaining them has always been the DBA domain. DBAs are focused on the database server performance now. As the number of instances grows, you will need more DBAs to handle the job of keeping all instances ship-shape daily. This is when you need to consider farm management as well.

Managing the database server farm is about managing and optimizing resources, cost, risk and inventory, planning, forecasting, reporting, and budgeting. Database Farm Management is focused on the medium- and long-term future, so it is usually done by senior DBAs and IT operations managers.

As an analogy, think of the difference between database farm management and instance management as the difference between managing public transport in a large city with managing a formula one race car team. The former is concerned with moving as many people as possible on buses, trams, and trains in a cost-efficient manner, while the latter is concerned with making one or two cars win the race at almost any cost.

We all experienced how the number of database instances kept growing (especially SQL Servers) in the last few years.

On October 15, 2019, Per Christopher held a webinar for the DBA Fundamentals virtual group in PASS about managing large database server farms. He compares how going from a small family farm to a large industrial farm compares to going from a handful of SQL servers to large database server farms, and looks at how you will need to update your mindset and tool set as the farms grows:

Monitoring databases in large,  distributed or hybrid environments

Monitoring database or SQL servers can in itself be a complex task - as we discuss in SQL Monitoring - 5 steps to full control. But what if you have a really complex environment, with multiple locations, restricted networks behind firewalls, high number of servers or a hybrid solution with some on-premise and some in a cloud? What are the issues, and how can you resolve them?

dbWatch Enterprise Manager 12

No matter what the service, no matter what the purpose, no matter what the product, the cloud is being touted as an answer to scalability by pretty much every major vendor. It started with data-storage, and rapidly moved to databases. In the past we were limited by the actual hardware that we owned. If we needed to expand on it, we needed to purchase more memory, and in many cases we needed to purchase new hardware as the old wore out, or was incapable of handling what we needed it to do.

It's an old problem. The more results you deliver, the more that are expected. The faster you provide them, the faster they are expected. Next thing you know, old methods don't work as well as they used to.

New demands require new workflows. On top of it new technologies are appearing making it seem like your old tools are no longer needed. Next thing you know the environment in which you work has changed. It is barely recognisable, and you might be afraid that you are no longer relevant.

As most DBAs know, security of data is one of the most difficult yet important tasks of maintaining a large estate of databases. It has kept more than one administrator up at night worrying about potential threats and pitfalls. With the growth of the information economy, not only is most information stored in databases, the value of that information has grown. As with anything with
value, the threats to security increases at a direct correlation to its worth.

Complexity is often a natural condition of most successful businesses. We build databases to handle complex data, to
maintain a layer of structure for important business information.

However, when building a database, or cluster of databases, typically the needs or requirements change over time. New
divisions or projects spring up. This is generally not a bad thing for a business or organisation. In most cases growth is good. However order to do this, without incurring huge amount of expense, often you a add these modules into existing databases rather than create new ones for different purposes.

There are many advantages to this approach; it makes accessing data easier if needed. However, in some (read: many) cases, you need to create new databases to handle different functions. One part of a business, such as vendor contract information, may have literally nothing to do with another, such as customer service records. So new databases are created. Maybe even the needs in one function, such as sales records increase beyond the capacity of the original
database they receive higher workloads than others.