vSphere Design Best Practices
Apply industry-accepted best practices to design
reliable high-performance datacenters for your
business needs
Brian Bolander
Christopher Kusek
P U B L I S H I N G
professional expertise distilled
vSphere Design Best Practices
Copyright © 2014 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: May 2014
Production Reference: 1200514
Published by Packt Publishing Ltd. Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78217-626-8
www.packtpub.com
Credits
Authors Brian Bolander
Christopher Kusek
Reviewers Andy Grant
Muhammad Zeeshan Munir
Prasenjit Sarkar
Commissioning Editor Rebecca Youe
Acquisition Editor Meeta Rajani
Content Development Editor Sharvari Tawde
Technical Editors Rosmy George
Manal Pednekar
Copy Editor
Laxmi Subramanian
Project Coordinator Melita Lobo
Proofreader Simran Bhogal
Indexers
Mariammal Chettiyar
Tejal Soni
Graphics Valentina Dsilva
Abhinash Sahu
Production Coordinator Alwin Roy
About the Authors
Brian Bolander
spent 13 years on active duty in the United States Air Force. A veteran of Operation Enduring Freedom, he was honorably discharged in 2005. He immediately returned to Afghanistan and worked on datacenter operations for the US Department of Defense (DoD) at various locations in southwest Asia. Invited to join a select team of internal consultants and troubleshooters responsible foroperations in five countries, he was also the project manager for what was then the
largest datacenter built in Afghanistan.
After leaving Afghanistan in 2011, he managed IT operations in a DoD datacenter in the San Francisco Bay Area. His team was responsible for dozens of multimillion dollar programs running on VMware and supporting users across the globe.
He scratched his adrenaline itch in 2011 when he went back "downrange", this time directing the premier engineering and installation team for the DoD in
Afghanistan. It was his privilege to lead this talented group of engineers who were responsible for architecting virtual datacenter installations, IT project management, infrastructure upgrades, and technology deployments on VMware for the entire country from 2011 to 2013.
Selected as a vExpert in 2014, he is currently supporting Operation Enduring Freedom as a senior virtualization and storage engineer.
Christopher Kusek
had a unique opportunity presented to him in 2013, to take the leadership position responsible for theater-wide infrastructure operations for the war effort in Afghanistan. Leveraging his leadership skills and expertise in virtualization, storage, applications, and security, he's been able to provide enterprise-quality service while operating in an environment that includes the real and regular challenges of heat, dust, rockets, and earthquakes.He has over 20 years' experience in the industry, with virtualization experience running back to the pre-1.0 days of VMware. He has shared his expertise with many far and wide through conferences, presentations, #CXIParty, and sponsoring or presenting at community events and outings whether it is focused on storage, VMworld, or cloud.
He is the author of VMware vSphere 5 Administration Instant Reference, Sybex, 2012, and VMware vSphere Performance: Designing CPU, Memory, Storage, and Networking for Performance-Intensive Workloads, Sybex, 2014. He is a frequent contributor to VMware Communities' Podcasts and vBrownbag, and has been an active blogger for over a decade.
A proud VMware vExpert and huge supporter of the program and growth of the
virtualization community, Christopher continues to find new ways to outreach
and spread the joys of virtualization and the transformative properties it has on individuals and businesses alike. He was named an EMC Elect in 2013, 2014, and continues to contribute to the storage community, whether directly or indirectly, with analysis and regular review.
He continues to update his blog with useful stories of virtualization and storage and his adventures throughout the world, which currently include stories of his times in Afghanistan. You can read his blog at http://pkguild.com or more likely catch him on Twitter; his twitter handle is @cxi.
When he is not busy changing the world, one virtual machine at a time or Facetiming
with his family on the other side of the world, he's trying to find awesome vegan
About the Reviewers
Andy Grant
works as a technical consultant for HP Enterprise Services. Andy's primary focus is datacenter infrastructure and virtualization projects across anumber of industries including government, healthcare, forestry, financial, gas
and oil, and international contracting. He currently holds a number of technical
certifications including VCAP4/5-DCA/DCD, VCP4/5, MCITP:EA, MCSE, CCNA,
Security+, A+, and HP ASE BladeSystem.
Outside of work, he enjoys backcountry camping, playing action pistol sports (IPSC), and spending time being a goof with his son.
Prasenjit Sarkar
is a senior member of the technical staff in VMware Service Provider Cloud R&D where he provides architectural oversight and technical guidance to design, implement, and test VMware's Cloud datacenters. You can follow him on Twitter at @stretchcloud.He is an author, R&D guy, and a blogger focusing on virtualization, cloud computing, storage, networking, and other enterprise technologies. He has more than 10 years' expert knowledge in R&D, professional services, alliances, solution engineering, consulting, and technical sales with expertise in architecting and deploying virtualization solutions and rolling out new technology and solution initiatives. His primary focus is on the VMware vSphere infrastructure and public cloud using VMware vCloud Suite. One of his other areas of focus is to own the entire life cycle of a VMware-based IaaS (SDDC), especially vSphere, vCloud Director, vShield Manager, and vCenter Operations.
He was one of the VMware vExperts in 2012 and 2013 and well known for his acclaimed virtualization blog http://stretch-cloud.info. He holds certifications from VMware, Cisco, Citrix, Red Hat, Microsoft, IBM, HP, and Exin. Prior to joining
VMware, he has served other fine organizations such as Capgemini, HP, and GE as a
solution architect and infrastructure architect.
www.PacktPub.com
Support files, eBooks, discount offers and more
You might want to visit www.PacktPub.com for support files and downloads related to your book.Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
TM
http://PacktLib.PacktPub.com
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.
Why Subscribe?
• Fully searchable across every book published by Packt • Copy and paste, print and bookmark content
• On demand and accessible via web browser
Free Access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.
Instant Updates on New Packt Books
Table of Contents
Preface 1
Chapter 1: Virtual Data Center Design
5
Virtual Data Center design principles 6
Best practices 7
Designing Virtual Data Center 8
VMware vCenter components 10
Choosing a platform for your vCenter server 11
Using the vCenter server appliance 12
Sizing your vCenter server 13
Choosing your vCenter database 14
vSphere clustering – HA and DRS 16
Host considerations 16
Network considerations 17
Storage considerations 19
Cluster considerations 19
Admission control 20
Summary 21
Chapter 2: Hypervisor Design
23
ESXi hardware design 23
CPU considerations 24
Memory and NUMA considerations 26
Virtual NUMA (vNUMA) considerations 28
Considerations for Java virtual machines (JVMs) 28
Network interface card considerations 29
Hypervisor storage components 31
Stateless host design 31
Scale-Up and Scale-Out designs 33
Table of Contents
[ ii ]
Chapter 3: Storage Design
37
Storage protocols 37
Fibre Channel (FC) 38
iSCSI 38
Fibre Channel over Ethernet (FCoE) 39
NFS 39
Virtual machine filesystems 40
VMFS 40 RDM 41 NFS 42
Designing storage 43
Local storage considerations 43 Datastore and cluster design 44 VMware storage features 45 Configurable storage features 46
SIOC 46
Configuring the multipathing path selection policy 47
Provisioning 47
Summary 48
Chapter 4: Network Design
49
Introducing vSphere switching 50
vNetwork Standard Switches (vSS) 50
vNetwork Distributed Switches (vDS) 50
Common networking design considerations 52
Best practices to implement in your design 53
Implementing Network I/O Control (NIOC) 53
Providing network redundancy, resiliency, and throughput 54
Physical switches and adapters 55
ESXi host hardware 56
Considerations when using IP storage 56
Separating your traffic 57
Validating your storage configurations 57
Summary 57
Chapter 5: Virtual Machine Design
59
Virtual machine resources and templates 59
Best practices for virtual machines and templates 60
Over-allocation can be a painful proposition 61
Physical to virtual migrations 61
Multi-vCPU considerations 62
Table of Contents
[ iii ]
Configuring shares 63
Limits 63 Reservations 63
Causes of virtual machine performance problems 64
CPU performance issues 64
Memory performance issues 65
Storage performance issues 66
Network performance issues 68
Summary 68
Chapter 6: Business Critical Applications
69
Special considerations for tier 1 applications 69
Starting simple 70
Workload profiles 71
Designing for Microsoft Exchange 71
Considering Microsoft SQL server 72
HA, DRS, and vMotion considerations 73
Resource considerations 75
Ensuring high availability and application resiliency 76
Enabling CPU Hot Add in vSphere client 77
The vCenter operations manager dashboard 78
Other business critical application resources 79
Summary 79
Chapter 7: Disaster Recovery and Business Continuity
81
The benefits of disaster avoidance 82
Designing backup strategies 83
Designing for RPO and RTO requirements 85
Choosing replication technologies 86
VMware vSphere Replication 86
Storage-based replication 87
Mixing replication solutions 88
Planning and testing the DR strategy 89
Summary 91
Appendix A: vSphere Automation Tools
93
Introducing VMware vCenter Orchestrator 93 Introducing VMware vCloud Automation Center 95
Unified IT service catalog 96
Advanced Service Designer 96
Multivendor interoperability 97
Designing for automation and XaaS 98
Table of Contents
[ iv ]
Appendix B: Certification Resources
99
VMware Certification Roadmap 100
Associate exams 101
Professional exams 101
Advanced professional exams 101
Expert-level defense 102
VMware vExpert program 102
Evangelist path 102
Customer path 103
VMware Partner Network (VPN) path 103
Recommended reading, blogs, and podcasts 103
Exam and study tips 104
Summary 104
Preface
Welcome to vSphere Design Best Practices, an easy-to-read guide full of hands-on examples of real-world design best practices. Each topic is explained and placed in the context of virtual datacenter design.
This book is all about design principles. It isn't about operations or administration; instead, we focus on designing virtual datacenters to meet business requirements and leveraging vSphere to get us to robust and highly available solutions.
In this book, you'll learn how to utilize the features of VMware to design, architect, and operate a virtual infrastructure using the VMware vSphere platform. Readers will walk away with a sense of all the details and parameters there are for a design that has been well thought out.
We'll examine how to customize your vSphere infrastructure to fit business needs and look at specific use cases for live production environments. Readers will become
familiar with new features in Version 5.5 of the vSphere suite and how these features can be leveraged in design.
Readers will walk away with confidence as they will know what their next steps are towards accomplishing their goals, whether that be a VCAP-DCD certification or a
sound design for an upcoming project.
What this book covers
Chapter 1, Virtual Data Center Design, gives you an insight into the design and architecture of the overall datacenter, including the core components of vCenter and ESXi.
Preface
[ 2 ]
Chapter 3, Storage Design, explains one of the most important design points of the datacenter—storage. This chapter focuses attention on the design principles related to the storage and storage protocols.
Chapter 4, Network Design, peels back the layers of networking by focusing on designing
flexible and scalable network architectures to support your virtual datacenter.
Chapter 5, Virtual Machine Design, covers what it takes to inspect your existing virtual machines and provides guidance and design principles to correctly size and deploy VMs.
Chapter 6, Business Critical Applications, breaks through the barriers and concerns associated with business critical applications, allowing business requirements to translate into a successful and stable deployment.
Chapter 7, Disaster Recovery and Business Continuity, considers the many parameters
that make up a DR/BC design, providing guidance to make sound decisions to
ensure a well-documented and tested disaster recovery policy.
Appendix A, vSphere Automation Tools, briefly discusses the value of automation tools
such as vCenter Orchestrator (vCO) and vCloud Automation Center (vCAC), their differences, and use cases for your design and implementation.
Appendix B, Certification Resources, gives guidance on the VMware certification roadmap and lists resources for the Data Center Virtualization (DCV) track.
What you need for this book
As this book is technical in nature, the reader should possess an understanding of the following concepts:
• Storage technologies (SAN, NAS, Fibre Channel, and iSCSI)
• Networking—both physical and virtual
• VMware vSphere products and technologies such as:
° Hypervisor basics
° vMotion and Storage vMotion
Preface
[ 3 ]
Who this book is for
This book is ideal for those who desire a better understanding of how to design virtual datacenters leveraging VMware vSphere and associated technologies. The typical reader will have a sound understanding of VMware vSphere fundamentals and would have been involved in the installation and administration of a VMware environment for more than two years.
Conventions
In this book, you will find a number of styles of text that distinguish between
different kinds of information. Here are some examples of these styles, and an explanation of their meaning.
Code words in text are shown as follows: "VMware Communities Roundtable podcast hosted by John Troyer (@jtroyer)."
New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: "You can
adjust your guests' vNUMA configuration on a per-VM basis via Advanced Settings
in order to adjust vNUMA to meet your needs."
Warnings or important notes appear in a box like this.
Tips and tricks appear like this.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.
Preface
[ 4 ]
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes
do happen. If you find a mistake in one of our books—maybe a mistake in the text or
the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this
book. If you find any errata, please report them by visiting http://www.packtpub. com/submit-errata, selecting your book, clicking on the erratasubmissionform link,
and entering the details of your errata. Once your errata are verified, your submission
will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.
Piracy
Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at copyright@packtpub.com with a link to the suspected pirated material.
We appreciate your help in protecting our authors, and our ability to bring you valuable content.
Questions
Virtual Data Center Design
The datacenter has quite simply become one of the most critical components to organizations today. Some datacenters, either owned by large enterprises or leased by large service providers, can contain great amounts of bleeding-edge technologies, tens of thousands of servers, huge network pipes from multiple carriers, and enough contingency mechanisms to keep their systems running for months in case of a disaster. Other datacenters can consist of just a small handful of servers and
off-the-shelf networking gear stuffed in a closet. Every organization is a bit different and has its own unique set of requirements in order to provide IT services to its users. As system administrators, engineers, and architects, we need to be able to take
those requirements, sometimes even search for or define them, and design a solution
to meet or exceed those requirements.
As datacenters have evolved, we've seen a paradigm shift towards virtualization. This is due to a number of factors including performance, availability, management, and recoverability. This book aims to help call attention to these factors and explains how to address them in your Virtual Data Center designs.
The Virtual Data Center has several key components such as compute, storage, networking, and management.
We will be covering the following topics in this chapter:
• Core design principles for the Virtual Data Center
• Best Practices for Virtual Data Center design
• How best practices change over time
• Virtual Data Center design scenarios
• vCenter design including the Linux-based vCenter Server Appliance (vCSA)
• vSphere clustering, HA, and DRS
Virtual Data Center Design
[ 6 ]
Virtual Data Center design principles
VMware vSphere is a leading platform for virtualization with many components that make up the VMware vCloud Suite including vCenter Server, ESXi hypervisor, and Site Recovery Manager (SRM). Through these products, the vSphere platform, with proper design characteristics, enables an IT infrastructure to provide services,
availability, flexibility, recoverability, and performance that its customers require.
Apart from knowledge of the aforementioned VMware products (or perhaps any third-party solutions being considered), there are a few key principles to get started with our designs. While there are numerous methodologies out there, this is a framework on which you can base your process consisting of three basic phases—Conceptual Design, Logical Design, and Physical Design, as shown in the following diagram:
Conceptual Design
Logical Design
Physical Design
It is important to remember that while the process is important, the focus of this book is about the design decisions. If you typically subscribe to a different framework or methodology, that is OK, as the design decisions discussed throughout the book will hold true regardless of your process and methodology in most cases.
First, create a Conceptual Design by gathering and analyzing business and application requirements, and then document the risks, constraints, and
Chapter 1
[ 7 ]
You certainly don't need to know the specifics at this point but you should have a vision and direction of the goal.
Next, building from the conceptual design, we'll begin formulating the Logical Design. To do this, we'll begin laying down the logical representations of the infrastructure components. Why just logical representations? Continuing with the above example, we don't know what servers we need or how many, we just know that we need servers. We don't know what storage array we need or what size, we just know that we need shared storage, and so on. We need to map each of the requirements to a logical solution within the complete design. This also begins to
flush out dependencies between components and services. While it would be great if
we could jump straight from the conceptual design right to the complete solution, we should be taking smaller steps towards our goal. This step is all about putting ideas down on paper and thinking through this Logical Design before we start procuring hardware and building up the datacenter by trial and error.
Finally, we transition our Logical Design into a Physical Design. While the previous steps allow us to be somewhat theoretical in our work, this phase absolutely requires
us to do our homework and put specific parameters around our design. For example,
we determine we need six servers with dual 8-core CPUs, 128 GB RAM, 15 TB of
array-based fiber channel attached storage, 4 x 10 Gbe NICs, 2 x 8 Gb Fiber Channel
HBAs, and so on. In most cases, you'll have this detail down to the manufacturer of each component as well. This physical design should consist of all the infrastructure components that will be required to support the requirements gathered in the Conceptual Design phase. For example, a complete Physical Design might include a vSphere Network Design, Storage Design, Compute Design, Virtual Machine (VM) Design, and Management Design. As you can imagine, the documentation for a physical design can get quite large and time consuming, but this is a critical step.
In summary, remember that the three design phases are additive and dependent upon one another. In order to have a successful Physical Design, both the Conceptual and Logical Designs must be solid. Document as much as is reasonable in your designs and even include reasoning behind choices. Many choices seem obvious while making them; however, when it is time to review and deploy a design months later, it may not be as obvious as to why certain choices were made.
Best practices
We hear the term Best practices everywhere but what does it really mean? Do we
always adhere to best practices? What happens to best practices as technology evolves, or as your business evolves? These are great questions and ones that many
Virtual Data Center Design
[ 8 ]
You may ignore best practices, strictly follow best practices, or employ a combination
of the two. The third scenario is the most common and should be your target. Why? The simplified answer is that there are certainly times when best practices aren't in
fact best for your design. A great example is the general best practice for vSphere where the defaults for just about everything in the environment are acceptable. For
example, take storage multipathing. VMware default is for fixed path and if best
practice is default, you may consider not modifying this; experience shows that often you'll see performance gains switching to Round Robin (see VMware KB1011340
for details). While changing this default may not be a best practice, then again, "Best
Practice" might not fit your needs in the real world. Choose what works for you and
the constraints of your environment.
Best practices also seem to have a shelf life much longer than intended. A favorite example is that the best size for vSphere datastore is 500 GB. While this sizing best practice was certainly true at one time, vSphere has improved how it handles datastore sizing using features such as SCSI Reservation, which enables much higher VM per datastore density. Since vSphere 5.0, we've been able to create 64 TB datastores and the best practice is more about evaluating parameters such as number of VMs, number of VMDKs per VM, required IOPS, SLAs, fault domains, and
restore capabilities. Many a times the backend disk capabilities will determine the ideal datastore size. Yet, many customers continue to use 500 GB as their standard datastore size. There are many more examples but the bottom line is that we should use best practices as a guideline and then do our homework to determine how to apply them to our designs.
Along with the best practices and design framework we've already discussed, it is also
important to define and adhere to standards throughout your Virtual Data Center.
This includes items such as naming conventions for ESXi hosts and VMs as well as
IP address schemes, storage layout, network configurations, and security policies.
Adhering to these standards will make understanding the design much easier as well as help as changes are introduced into the environment after deployment.
Designing Virtual Data Center
As previously mentioned VMware's Virtual Data Center (vDC) is composed of several key components: vCenter, ESXi hypervisor, CPU, storage, networking, and virtual machines. Many datacenters will have additional components but these form the foundation of typical vDC. Note that it is possible to operate, albeit in a limited
capacity, without vCenter in the management layer. We find that it is really the core
Chapter 1
[ 9 ]
Our vDCs will also consist of several constructs that we will leverage in our designs to provide organization and operability. These include the following:
• Datacenters
• Clusters
• Resource pools
• Folders
• Tags
By using these constructs, we can bring consistency into complex designs. For example, by using folders and tags, we can organize VMs, hosts, and other objects so
we can easily find them, report on them, or even perform operations such as backup
based on folders or tags.
Tags are a new feature in vSphere 5.1 and above that work much like tagging in other applications. It allows us to put arbitrary tags on objects in our vDC for organizational purposes. Folders present a challenge when we have objects that could belong in multiple folders (which isn't possible). Many administrators use a combination of folders and tags to both organize their inventory as well manage it.
The following screenshot is a specific example of how folders can be used to organize
our vDC inventory:
It is important to define a standard folder hierarchy as part of your design and
Virtual Data Center Design
[ 10 ]
Another facet to our designs will be how we might employ multiple vCenters and how we manage them. Many organizations utilize multiple vCenters because they
want to manage Production, QA, and Test/Dev, separately. Others have vCenters
distributed geographically with a vCenter managing each local infrastructure. In any case, with vSphere 5.5, multiple vCenter management has become much easier. We have what is called Linked Mode as an option. This allows us to provide a single pane of access to resources connecting to multiple vCenters—think shared
user roles/groups and being able to access objects from any vCenters that are linked
through a single console. However, with the vSphere web client included with vSphere 5.1 and above, we are able to add multiple vCenters to our view in the client. So, if your requirements are to have less roles and groups to manage, then
Linked Mode may be the best route. If your requirements are to be able to manage multiple vCenters from a single interface, then the standard vSphere web client should meet your needs without adding the complexity of Linked Mode.
Next, we have Single sign-on (SSO). This is a new feature introduced in vSphere 5.1
and improved in vSphere 5.5. Previously, SSO had a difficult installation process and some difficult design decisions with respect to HA and multisite configurations. With
SSO in vSphere 5.5, VMware has consolidated down to a single deployment model,
so we're able to much more easily design HA and multisite configurations because
they are integrated by default.
Finally, there is the subject of SSL certificates. Prior to vSphere 5.0, it was a best practice (and a difficult task) to replace the default self-signed certificates of the vSphere components (vCenter, ESX, and so on) with CA-signed certificates. That
remains the case, but fortunately, VMware has developed and released the vCenter
Certificate Automation tool, which greatly reduces the headaches caused when
attempting to manually generate and replace all of those SSL certificates. Although not a steadfast requirement, it is certainly recommended to include CA-signed
certificates in your designs.
VMware vCenter components
There are four primary components to vCenter 5.1 and 5.5:
• Single sign-on: This provides identity management for administrators and applications that interact with the vSphere platform
• vSphere web client: This is the service that provides the web-based GUI for administrators to interact with vCenter and the objects it manages
Chapter 1
[ 11 ]
• vCenter server: This is the core service of vCenter, which is required by all other components
There are also several optional components and support tools, which are as follows:
• vSphere client: This is the "legacy" C#-based client that will no longer be available in the future versions of vSphere (although still required to interact with vSphere Update Manager)
• vSphere Update Manager (VUM): This is a tool used to deploy upgrades and patches to ESXi hosts and VMware tools and virtual hardware version updates to VMs
• vSphere ESXi Dump Collector: This is a tool used mostly to configure
stateless ESXi hosts (that is, have no local storage) to dump VMkernel memory to a network location and then pull those logs back into vCenter
• vSphere Syslog Collector: This is a tool similar to the Dump Collector that allows ESXi system logs to be redirected to a network location and be accessed via vCenter
• vSphere Auto Deploy: This is a tool that can provision ESXi hosts over the network and load ESXi directly into memory allowing for stateless hosts and
efficient provisioning
• vSphere Authentication Proxy: This is a service that is typically used with Auto Deploy so that hosts can be joined to a directory service, such as MS
Active Directory, without the need to store credentials in a configuration file
At the onset of your design, these components should be accounted for (if used) and integrated into the design. For small environments, it is generally acceptable to combine these services and roles on the vCenter while larger environments will generally want to separate these roles as much as possible to increase reliability by reducing the fault domain. This ensures that the core vCenter services have the resources they need to operate effectively.
Choosing a platform for your vCenter server
Now that we've reviewed the components of vCenter, we can start looking at
making some design decisions. The first design decision we'll need to make relative
to vCenter is whether you'll have a physical or virtual vCenter. VMware's own recommendation is to run vCenter as a VM. However, just as with best practices,
we should examine and understand why before making a final decision. There are
valid reasons for having a physical vCenter and it is acceptable to do. However, if we make the appropriate design decisions, a virtual vCenter can benefit from all the
Virtual Data Center Design
[ 12 ]
One way to mitigate risks is to utilize a Management Cluster (sometimes called a management pod) with dedicated hosts for vCenter, DNS, Active Directory,
monitoring systems, and other management infrastructure. This benefits us in a few
ways such as being able to more easily locate these types of VMs if vCenter is down. Rather than connecting to multiple ESXi hosts via Secure Shell (SSH) or using the vSphere Client to connect to multiple hosts in order to locate a domain controller to be manually powered on, we're able to identify its exact location within our management pod. Using a management cluster is sometimes not an option due to the cost of a management pod, separate vCenter server for management, and the cost of managing a separate cluster. In smaller environments, it isn't such a big deal because we don't have many hosts to go searching around for VMs. But, in general,
a dedicated cluster for management infrastructure is recommended.
A second way to mitigate risks for vCenter is to use a product called vCenter Heartbeat. VMware vCenter Heartbeat provides automated and manual failover and failback of your vCenter Server and ensures availability at the application and service layer. It can restart or restore individual services while monitoring and protecting the Microsoft SQL Server database associated with vCenter Server, even if it's installed on a separate server. For more information on vCenter Heartbeat, you can go to the product page at http://www.vmware.com/products/vcenter-server-heartbeat/.
In general, if choosing a virtual vCenter, do everything possible to ensure that the vCenter VM has a high priority with resources and HA restarts. An option would be to set CPU and memory resource shares to High for the vCenter VM. Also, do
not disable DRS or use host affinity rules for the vCenter VM to try to reduce its movement within the vDC. This will negate many of the benefits of having a
virtual vCenter.
Using the vCenter server appliance
Traditionally, this decision hasn't really required much thought on our part. The vCenter Server Appliance (vCSA or sometimes called vCVA or vCenter Virtual Appliance) prior to vSphere 5.5 was not supported for production use. It also had a
limit of managing five hosts and 50 VMs. However, with vSphere 5.5, vCSA is fully
supported in production and can manage 1000 hosts and 10000 VMs. See vSphere
Chapter 1
[ 13 ]
So why would we want to use an appliance versus a full Windows VM? The appliance is Linux-based and comes with an embedded vPostgres, which will support up to 100 hosts and 3000 VMs. In order to scale higher, you need to use an external Oracle database server. Environments can save on Windows and MS SQL licensing by leveraging the vCSA. It is also a much simpler deployment. There are a few caveats, however, in that VMware Update Manager (VUM) still requires a separate Windows-based host with an MS SQL database as of vSphere 5.5. If you are running Horizon View, then you'll need a Windows server for the View Composer service and a similar situation exists if you are running vCloud Director. The general idea, though, is that the vCSA is easier to manage and is a purpose-built appliance. While vCSA may not be for everyone, but I would encourage you to at least consider it for your designs.
Sizing your vCenter server
If you're using the vCSA, this is less of an issue but there are still guidelines for disk space and memory as shown in the following table. Note that this only applies to vCSA 5.5 and above as the vCSA prior to 5.5 is only supported with a maximum of
five hosts and 50 VMs.
VMware vCenter Server Appliance Hardware
Requirement
Disk storage on the host machine
The vCenter Server Appliance requires at least 7 GB of disk space, and is limited to a maximum size of 80 GB. The vCenter Server Appliance can be deployed with thin-provisioned virtual disks that can grow to the maximum size of 80 GB. If the host machine does not have enough free disk space to accommodate the growth of the vCenter Server Appliance virtual disks, vCenter Server might cease operation, and you will not be able to manage your vSphere environment.
Memory in the VMware vCenter Server Appliance
• Very small (≤ 10 hosts, ≤ 100 virtual machines): This has a size of at least 4 GB.
• Small (10-100 hosts or 100-1000 virtual machines): This has a size of at least 8 GB.
• Medium (100-400 hosts or 1000-4000 virtual machines): This has a size of at least 16 GB.
• Large (≥ 400 hosts or 4000 virtual machines): This has a size of at least 24 GB.
Virtual Data Center Design
[ 14 ]
If you are choosing a Windows-based vCenter, you need to size your machine appropriately based on the size of your environment. This includes CPU, memory,
disk space, and network speed/connectivity.
The number of clusters and VMs within a vCenter can absolutely affect performance. This is due to DRS calculations that must be made and more objects mean more calculations. Take this into account when estimating resource requirements for your vCenter to ensure performance and reliability.
It is also recommended to adjust the JVM (Java Virtual Machine) heap settings for the
vCenter Management Web service (tcServer), Inventory Service, and Profile-driven
Storage Service based on the size of deployment. Note that this only affects vCenter 5.1 and above.
The following VMware KB articles have up-to-date information for your specific
vCenter version and should be referenced for vCenter sizing:
vCenter version VMware KB article number
5.0 2003790
5.1 2021202
5.5 2052334
Choosing your vCenter database
The vCenter database is the location where all configuration information, along
with some performance, task, and event data, is stored. As you might imagine, the vCenter DB can be a critical component to your virtual infrastructure. But it also can be disposable—although this is becoming a much less adopted strategy than it was a couple of years ago. In some environments that weren't utilizing features like the Virtual Distributed Switch (VDS) and didn't have a high reliance on the stored event data, we saw that vCenter could be quickly and easily reinstalled from scratch without much of an impact. In today's vDC's, however, we see more widespread use of VDS and peripheral VMware technologies that make the vCenter DB much more
important and critical to keep backed up. For example, the VDS configuration is housed in the vCenter DB, so if we were to lose the vCenter DB, it becomes difficult
Chapter 1
[ 15 ]
There are two other components that require a DB in vCenter and those are VUM and SSO. SSO is a new component introduced in vSphere 5.1 and then completely rewritten in vSphere 5.5. If you haven't deployed 5.1 or 5.5 yet, I'd strongly
encourage you to go to 5.5 when you're ready to upgrade due to this re-architecture of SSO. In vSphere 5.1, the SSO setup requires manual DB configuration with SQL authentication and JDBC SSL, which caused many headaches and many support tickets with VMware support during the upgrade process.
We have the following choices for our vCenter DB:
• MS SQL Express
• MS SQL
• IBM DB2
• Oracle
The different versions of those DBs are supported depending on the version of vCenter you are deploying, so check the VMware Product Interoperability Matrixes located at http://partnerweb.vmware.com/comp_guide/sim/interop_ matrix.php.
The MS SQL Express DB is used for small deployments—maximum of five hosts and
50 VMs—and is the easiest to set up and configure. Sometimes, this is an option for dev/test environments as well.
Next, we have probably the most popular option, that is, MS SQL. MS SQL can either be installed on the vCenter itself, but it is recommended to have a separate server dedicated to SQL. Again, sometimes in smaller deployments, it is OK to install the SQL DB on the vCenter server, but the general best practice is to have that dedicated database server for better protection, flexibility, and availability. With vCenter 5.5, we now have broader support of Windows Failover Clustering (formerly Microsoft Cluster Services or MSCS), which enhances our options for the availability of the vCenter DB.
Last, we have Oracle and IBM DB2. These two options are common for organizations that already have these databases in use for other applications. Each database
type has its own benefits and features. Choose the database that will be the easiest
to manage while providing the features that are required. In general, any of the database choices are completely capable of providing the features and
Virtual Data Center Design
[ 16 ]
vSphere clustering – HA and DRS
Aside from vCenter itself, HA and DRS are two features that help unlock the true potential of virtualization. This combination of automatic failover and workload
balancing result in more effective and efficient use of resources. The mechanics of
HA and DRS are a full topic in and of itself, so we'll focus on how to design with HA and DRS in mind.
The following components will be covered in this section:
• Host considerations
• Networking design considerations
• Storage design considerations
• Cluster configuration considerations • Admission control
Host considerations
Providing high availability to your vSphere environment means making every reasonable effort to eliminate any single point of failure (SPoF). Within our hosts, we can make sure we're using redundant power supplies, ECC memory, multiple
I/O adapters (such as NICs and HBAs), and using appropriate remote monitoring
and alerting tools. We can also distribute our hosts across multiple racks or blade chassis to guard against the failure of a rack or chassis bringing down an entire cluster. So not only is component redundancy important, but it is also important to look at the physical location.
Some other items related to our hosts are using identical hardware. Although not always possible, such as hosts that have different lifecycles, we should make an effort to be consistent in our host hardware within a cluster. Aside from
simplifying configuration and management, hardware consistency reduces resource
fragmentation. For example, if we have hosts with Intel E5-2640 CPUs and E5-2690 CPUs, the cluster will become unbalanced and result in fragmentation. HA prepares for a worst-case scenario and plans for the largest host in a cluster to fail resulting in
more resources being reserved for that HA event. This results in a lower efficiency in
resource utilization.
The next consideration with HA is the number of hosts in your design. We typically
look for an N+1 configuration for our cluster. N+1 represents the number of hosts we
Chapter 1
[ 17 ]
In some cases, it can be acceptable to actually reserve a host or hosts for failover. This is truly a hot spare scenario where those designated hosts sit idle and are used only in the case of an HA event. This is an acceptable practice, but it isn't the most efficient use of resources.
VMware vSphere 5.5 supports clusters of up to 32 hosts and it is important to understand how HA reserves resources based on cluster size. HA essentially reserves an entire hosts' worth of resources and then spreads that reservation across all hosts in the cluster. For example, if you have two hosts in a cluster, 50 percent of the total cluster resources are reserved for HA. To put it another way, if one host fails, the other host needs to have enough resource capacity to service the entire workload. If you have three hosts in a cluster, then 33 percent of the resources are reserved in the cluster for HA. You may have picked up on the pattern that HA reserves N
1
)
)
resources for the cluster where N is equal to the number of hosts in the cluster. As we increase the number of hosts in a cluster, we see a smaller percentage of resources reserved for HA that result in higher resource utilization. One couldargue that larger clusters increase complexity but that is negated by the benefits of
higher utilization. The important takeaway here is to remember that these reserved resources need to be available to the hosts. If they are being used up by VMs and an HA event occurs, it is possible that VMs will not be restarted automatically and you'll experience downtime.
Last, with respect to hosts, is versioning. It is absolutely recommended to have all hosts within a cluster at the same major and minor versions of ESXi, along with the same patch level. Mixed-host clusters are supported but because there are differences in HA, DRS, and other features, functional and performance issues can result. If different host versions are required, it is recommended to separate those hosts into different clusters, that is, a cluster for 5.0 hosts and a cluster for 5.5 hosts.
For a VMware HA percentage Admission Control Calculator see
http://thinkingloudoncloud.com/calculator/vmware-ha-admission-control-calculator/.
Network considerations
Specific to HA, there are several design considerations regarding the network.
Even though the dependency HA had on DNS has been removed in vSphere 5.0 and above, it is still a good practice to ensure all hosts and VMs can be resolved through
Virtual Data Center Design
[ 18 ]
STP is a protocol that runs on network switches to prevent network loops.
For example, if a host is rebooted, STP takes approximately 50 seconds to recalculate the topology on its connected switch ports, and while that calculation is occurring the host does not have network connectivity. This can cause HA to report the host as dead and VMs will not start up on that host. This can also trigger an isolation response due to the datastore heartbeat. In either case, this can be prevented by enabling PortFast on those ports on the connected switches. Some switch vendors may call the feature something different, but the idea of PortFast is that STP will
allow traffic to pass on those ports while it completes its calculations.
HA relies on the management network of the hosts. This brings with it several design considerations. Each host should be able to communicate with all other hosts' management interfaces within a cluster. Most of the time, this is not an issue because our network designs typically have all of our hosts' management VMkernel ports on the same layer 2 subnet, but there are scenarios where this is not the case. Therefore, it is important to make sure all hosts within a cluster can communicate properly to ensure proper HA functionality and prevent network partitions. Along these same lines, we should be designing our networks so that we have the fewest possible number of hops between hosts. Additional hops, or hardware segments, can cause higher latency and increase points of failure.
Because of the criticality of the management VMkernel port for proper HA functionality, it is imperative that we design in redundancy for this network. This means having more than one physical NIC (pNIC) in our hosts assigned to this network. If there is a failure in the management network, we still may get host isolation since HA won't have a means to test connectivity between vSphere servers.
When standardizing your network configuration, you should use consistent names
Chapter 1
[ 19 ]
Storage considerations
HA, as previously mentioned, also relies on datastore heartbeats to determine if an HA event has occurred. Prior to vSphere 5.0, we only had the network to determine whether a host had failed or become isolated. Now we can use datastores to aid in that determination enhancing our HA functionality. Datastore connectivity is inherently important—after all, our VMs can't run if they can't connect to their
storage, right? So, in the case where we are utilizing a storage array, practices such
as multipathing continue to be important and we need to make sure our hosts have multiple paths to our storage array whether that is iSCSI, NFS, FC, or FCoE.
vCenter will automatically choose two datastores to use for heartbeats. However, we do have the capability to override that choice. A scenario where this makes sense is if we have multiple storage arrays. It would be recommended to have a heartbeat
datastore on two different arrays. Note that we can have up to five heartbeat
datastores. Otherwise, the algorithm for choosing the heartbeat datastores is fairly robust and normally does not need to be tampered with. Another consideration is that if we are using all IP-based storage, we should try to have separate physical network infrastructure to connect to that storage to minimize the risk of the management network and storage network being disrupted at the same time. Last, wherever possible, all hosts in a cluster should have access to the same datastores to maximize the ability of isolated hosts to communicate via the datastore heartbeat.
Cluster considerations
Of the many configurations we have within a cluster for HA, Host Isolation Response seems to be often neglected. While we previously talked about how to make our management networks robust and resilient, it is still important to decide how we want our cluster to respond in the event of a failure. We have the following three options to choose from for the Host Isolation Response:
• Leave Powered On: VMs will remain running on their host even though the host has become isolated
• Power Off: VMs will be forcibly shutdown as if pulling the power plugs from a host
• Shutdown: Using VMware tools, this option attempts to gracefully
Virtual Data Center Design
[ 20 ]
In many cases, the management network may be disrupted but VMs are not affected. In a properly architected environment, host isolation is a rare occurrence,
especially with a fiber channel SAN fabric as datastore heartbeats will be
out-of-band of the network. If the design requires IP storage, which uses the same network infrastructure as the management network, then the recommended option is to shut down during isolation as it is likely that a disruption in the management network will also affect access to the VM's datastore(s). This ensures that another host can power on the affected VMs.
Another HA feature is VM and application monitoring. This feature uses VMware tools or an application agent to send heartbeats from the guest OS or application to the HA agent running on the ESXi host. If a consecutive number of heartbeats are lost, HA
can be configured to restart the VM. The recommendation here is to utilize this feature
and decide ahead of time how tolerant of heartbeat failures you wish to be.
Admission control
Admission Control (AC) is another great feature of HA. Because AC can prevent VMs from being powered on during a normal operation or restarted during an HA
event, many administrators tend to turn AC off. HA uses AC to ensure that sufficient
resources exist in a cluster and are reserved for VM recovery during an HA event. In other words, without AC enabled, you can overcommit your hosts to the point
that if there was to be an HA event, you may find yourself with some VMs that are
unable to power on. Let's take another look at a previous example where we have three hosts in a cluster. HA reserves 33 percent of that cluster for failover. Without AC enabled, we can still utilize that 33 percent reserve which puts us at risk. With AC, we are unable to utilize those reserved resources, which mitigates that risk of overprovisioning and ensures we'll have enough resources to power on our VMs during an HA event. When Admission Control is enabled, we can choose from the following three policies:
• Host Failures Cluster Tolerates (default): This reserves resources based on the number of hosts for which we want to tolerate failures. For example, in a 10 host cluster, we could want to tolerate two host failures, so instead of the default one host failures the cluster tolerates (remember the N
1
)
)
formula), we would be allowing two host failures to tolerate in the cluster for failover.Chapter 1
[ 21 ]
• Percentage of Cluster Resources Reserved: This reserves the specified
percentage of CPU and memory resources for failover. If you have a cluster
where VMs have significantly different CPU and memory reservations
or hosts with different CPU and memory capacities, this policy is
recommended. This setting must be revisited as the cluster grows or shrinks.
• Specify a Failover Host: This designates a specific host or hosts as a
hot spare.
The general recommendation is to use the percentage of cluster resources policy for
Admission Control because it offers maximum flexibility. Design the percentage
reserved to represent the number of host failures to be tolerated. For example, in a four host cluster, if you want to tolerate one host failure your percentage of failures tolerated is 25 percent, and if you wanted to tolerate two host failures it is 50 percent. If there will be hosts in the cluster of different sizes, be sure to use the largest host(s) as your reference for determining the percentage to reserve. Similarly, if the Specify a Failover Host policy is used, we must specify hosts that have CPU and memory equal to or larger than the largest nonfailover host in the cluster.
If the Host Failures Cluster Tolerates policy is to be used, then make every reasonable attempt to use similar VM resource reservations across the cluster. This is due to the slot size calculation mentioned earlier.
Summary
In this chapter, we learned about the design characteristics of the Virtual Data Center and the core components of the VMware vSphere platform. We also discussed best
practices and design considerations specifically around vCenter, Clustering, and HA.
Hypervisor Design
This chapter begins a series of chapters regarding the architecture of specific virtual
datacenter components. The hypervisor is a critical component and this chapter will examine various best practices for hypervisor design.
We will cover the following topics in this chapter:
• ESXi hardware design including CPU, memory, and NICs
• NUMA, vNUMA, and JVM considerations
• Hypervisor storage components
• Stateless host design
• Scale-Up and Scale-Out designs
ESXi hardware design
Over the past 15 years we've seen a paradigm shift in server technology from mainframe and proprietary systems such as PowerPC, Alpha, and Itanium to commodity x86 architecture. This shift, in conjunction with x86 virtualization,
has yielded tremendous benefits and has allowed us both flexibility and
standardization with our hardware designs. If we take a look at the vSphere
Hardware Compatibility List (HCL), we only see two CPU manufacturers: Intel and AMD (Advanced Micro Devices). The VMware HCL can be found at http://www.vmware.com/resources/compatibility/search.php.
Hypervisor Design
[ 24 ]
CPU considerations
Naturally, one of the first decisions we find ourselves making when designing
our hosts is which chip manufacturer and which CPU will go into our hosts. As mentioned in Chapter 1, Virtual Data Center Design, standardization goes a long way and our host designs are no different. However, there are certainly instances in which we may choose different CPUs for our hosts. It remains a best practice
to keep the same CPU model consistent throughout a cluster. If we find ourselves
in a situation where we need different CPU models within a cluster, such as adding new hosts to an existing cluster, we have a great feature called Enhanced vMotion Compatibility (EVC). As long as we're running CPUs from the same chip manufacturer (that is all Intel or all AMD), we can mask CPU features to a lowest common denominator to ensure vMotions can occur between different chip models. For example, without EVC, we could not vMotion a VM from a host with an Intel Xeon 5500-series CPU to a host with the latest Intel Xeon E5-2600 CPU. With EVC this is completely possible and works very well. For more information on the EVC requirements, see VMware KB 1003212.
CPUs have many parameters to look at when we're choosing the best one(s) for our host design. The following parameters are the most common factors in determining the appropriate CPU for a host:
• Cores/threads • Clock speed
• Cache
• Power draw (watts)
• vSphere feature support
• Memory support
• Max CPUs supported (2-way, 4-way, 6-way, and so on)
Each of these parameters can be integral to a design while we may not use the others. We still need to consider the effects each one may have on our designs. Provided we have a good conceptual design, and thus proper requirements, we should be able to determine how many CPU resources we will need. If we have a requirement for eight cores per socket or 12 GHz CPU per host or maybe more commonly a density
of 60 VMs per host is specified. We can then profile the workload to determine the
proper range of CPUs to use. A range is used because cost is typically always an
influencing factor. Initially, we may come up with a range and then research cost to
Chapter 2
[ 25 ]
Continuing with the example where we require 60 VMs per host, there is an
important metric to look at when considering consolidation ratios: the ratio of vCPUs to pCPUs denoted as vCPU:pCPU. A vCPU is a virtual CPU that is presented to a VM while the pCPU represents a physical CPU in the host system. Furthermore, with technologies such as hyperthreading, pCPU can represent a logical processor. For instance, an Intel Xeon E5-2640 CPU has eight cores and hyperthreading enabled, which yields 16 pCPUs because each core can execute two threads. Each thread represents a logical processor (pCPU) and thus 8 x 2 = 16 logical processors or 16 pCPUs. VMware Best Practices for Performance with Hyper-threading can be read about in more detail at http://www.vmware.com/pdf/Perf_Best_Practices_ vSphere5.0.pdf.
Sometimes we have project requirements that allow us to start with a specific goal of
an overall vCPU:pCPU ratio or specific ratios for different parts of the infrastructure.
From there, we can choose a CPU and server platform that can accommodate those requirements. Let's get back to that 60 VMs per host example, and for this we will assume that these are all general purpose VMs with no special requirements and an average of 1.5 vCPUs per VM because of a mixture of one vCPU and two vCPU machines. Our design targets a vCPU: pCPU ratio of 2:1. The math is as follows: 1.5 x 60 = 90 vCPUs. Therefore, we'll require at least 45 pCPUs per host to achieve our 2:1 target (90:45). An eight-socket server with six-core CPUs would yield 48 pCPUs and
would accommodate the needs for this design. A quad socket configuration with
12-core CPUs would also yield 48 pCPUs. There are many ways to make the math
work and other considerations such as using high availability configurations,
additional capacity for growth that we will discuss as we continue our design.
For general purpose predictable workloads that don't have special requirements (such as SQL, Exchange, SharePoint, and VDI), we typically see a vCPU:pCPU ratio of 2:1 to 4:1 inclusive with higher ratios for test and development environments. But this is certainly a situation where your mileage may vary, so you need to know your applications and workload in order to intelligently choose the appropriate ratio for your environment.
Some of the other factors, such as cache and clock speed, are a bit more
Hypervisor Design
[ 26 ]
Memory and NUMA considerations
While CPUs provide the raw processing power for our VMs, memory provides the capacity for our applications and workload to run.
Also important are High Availability (HA) admission controls. If strict admission controls are enabled, then guests that violate HA controls won't come back online after a failure. Instead, by employing relaxed admission controls, you'll see
performance degradation but won't have to deal with the pain of HA causing VMs to power off and not come back. By considering the speed, quantity, and density of our host-based RAM, we're able to avoid those issues and provide predictable performance to our virtual infrastructure. This means that your design depends on sizing memory and calculating HA minimums to ensure you don't cause problems.
At the highest level, the total amount of host memory is pretty straightforward. As we take into consideration our workload and application requirements, we can start by simply dividing the total amount of RAM required by the number of hosts to get the amount of memory we want in each host. Just as before, there are tools to help us determine the amount of memory we need. VMware Capacity Planner is one of
those tools and is available to VMware partners. VMware has some free flings such
as vBenchmark and ESX System Analyzer. Other tools such as Dell Quest's vFoglight and SolarWinds Virtualization Manager are commercially available and can provide detailed reporting and recommendations for properly sizing your virtual environment.
In most cases we're designing an N+1 architecture, we'll then add an extra host with that amount of RAM into the cluster. For example, if our overall workload requires one TB of RAM, we may design for four hosts each with 256 GB of RAM. To achieve the proper failover capacity required for HA and our N+1 architecture, we'll add a
fifth host for a total of 1.25 TB.
Chapter 2
[ 27 ]
Each NUMA node is situated the closest to its associated pCPU and results in a reduced time to access that memory. This reduces memory access contention as only a single pCPU can access the memory space at a time. pCPUs may simultaneously access their NUMA nodes resulting in dramatically increased performance over non-NUMA, or UMA systems. The following diagram shows the NUMA architecture:
Node 0
Intersocket
connection Node 1
L
ocal
Memor
y
node
0
L
ocal
Memor
y
node
1
CPU CPU
Remote access Local
access
1 2
3 4
1 2
3 4
NUMA Architecture
Some server manufacturers include settings in the server's BIOS, which will impact NUMA. One such setting is called Node Interleaving and should be disabled if we want to take advantage of NUMA optimizations. For more information, have a look at the White Paper from VMware at https://www.vmware.com/files/pdf/ techpaper/VMware-vSphere-CPU-Sched-Perf.pdf.
Hypervisor Design
[ 28 ]
Virtual NUMA (vNUMA) considerations
ESXi hosts create a virtual NUMA topology that closely matches the server's
underlying physical NUMA layout. This vNUMA topology ensures that guests have optimal access to compute and memory resources. vNUMA for VMs is calculated for the host on which the VM starts and will not adjust if a vMotion operation takes the VM to another host; this is another good reason to employ hosts with the same
hardware configurations. Additionally, vNUMA ensures a guest's memory and
processing power is provisioned and in line with the physical NUMA layout that lives underneath, even if that topology spans more than one NUMA node.
vNUMA has a few caveats to bear in mind. Your VMs have to be VM Version 8 and running on vSphere 5.0 (or higher). Guests with eight vCPUs enjoy automatic vNUMA
configuration. It's very likely that monster VMs with many vCPUs will access multiple
NUMA nodes when running, so this makes sense from a default settings point of view. Defaults aren't always best when large VMs have unique requirements. You can adjust
your guests' vNUMA configuration on a per-VM basis via Advanced Settings in order to adjust vNUMA to meet your needs. Do this to meet a particular VM's demands, or when tailoring vNUMA for better performance in vApps with multiple guests.
Considerations for Java virtual
machines (JVMs)
When designing JVMs, some special considerations exist. Bear these in mind while planning and deploying your Java VMs.
When determining the number of vCPUs to allocate to a new JVM, test performance until you arrive at a number of vCPUs that provide the performance you need under varying loads. If you have access to a testbed, do the performance tests in that environment. If you don't have access to a lab environment or you can't test in your production environment, be prepared to adjust the number of vCPUs allocated to the JVM. You want to arrive at a vCPU allocation that doesn't inhibit the JVM under a variety of loads. If your Java code runs multiple garbage collector threads, ensure that your vCPU allocation matches the number of threads.
Chapter 2
[ 29 ]
Lastly, JVM processes should be configured on a one-for-one VM basis. Don't load up one JVM with multiple Java processes. You'll avoid resource contention and make your life much easier when monitoring your JVMs.
Network interface card considerations
While CPU and memory are integral components of our ESXi hosts, we won't have an environment without network connectivity. NICs provide network connectivity to our hosts, VMs, and may also provide storage connectivity if our designs that utilize iSCSI, NFS, or fiber channel over ethernet (FCoE). In this section, we'll examine some
scenarios and discuss guidelines for quantity, type, and configuration of NICs.
Let's start by looking at the most common deployments of 1 Gbps Ethernet (1 GbE)
NICs. Dedicating NICs to different traffic types is typically ideal. In other words, at a minimum, we see a pair of NICs for management traffic, vMotion, Fault Tolerance
(FT) (if used), IP Storage (if used), and VMs for a total of ten NICs. This may seem like quite a few, but quad-port NICs (NIC cards with four 1 GbE ports onboard) are quite common and cost effective nowadays. If we're not using FT- or IP-based storage, that immediately drops us down to six NICs. In either case, it is a good idea to have multiple physical NIC cards in the system. This, in conjunction with using NIC teaming, can provide protection against the failure of an entire NIC card.
NIC teaming is used to provide resiliency to our hosts' network uplinks. For instance, to provide high availability to our hosts' management network, we can bond multiple NICs to that network. If one NIC fails, we will still have management network functionality as long as one NIC remains operational.
To fully realize the benefits of NIC teaming with multiple NICs, we need to be sure
Hypervisor Design
[ 30 ]
In general, we'll see a minimum of six to ten NICs in an ESXi host with NIC teaming being used to spread bandwidth and connectivity across multiple physical NIC cards for the highest possible resiliency. With respect to bandwidth, there are tools that can
be used to monitor and/or estimate the required bandwidth needed to support our
workload. VMware's Capacity Planner is one of those tools but there are other tools. Based on the output of those tools, we can determine the quantity of NICs required
for VM connec
