When we are asked to audit a server room or data centre one of the first things, we want to establish are the design standards and management systems the facility should conform to. The answers to this question can vary dependent upon the size of the IT facility and to some extent the industries they serve.
Hyperscale and colocation data centres will typically require auditing to specific standards and levels of availability. Smaller on-premise enterprise data centres and server rooms may be more concerned with how best to solve a specific problem such as a critical cooling issue, how to implement greater power resilience and/or improver overall security and business continuity planning.
Data centres have been around since the early 1950s, when computing companies such as IBM deployed their mainframe computers. At that time there were no specific standards for data centre design.
Over the decades the industry evolved with a reliance on standards for networking and telecommunications. In Europe, the principle standard for data centre design has been EN 50173, covering Information Technology and Generic Cabling Systems. In the US ANSI/TIA-942, the US standard for Telecommunications Infrastructure for Data Center, guided design and best practice. The issue for both standards is that they are cabling-based and whilst they have been updated, they do not specifically mention critical power and cooling system infrastructures – a main criticism of the latest edition of EN 50173-5:2018 which now covers Data Centre Spaces.
European Standards (EN) are documents ratified by three European Standards Organisations including CEN, CENELEC and ETSI. Recognising the need for a specific data centre standard has given rise to the EN 50600 series of standards for best practice and design standards in data centres, including power, cooling, telecommunications, and security (including fire suppression). The standard also covers operations and management and provides recommendations for energy usage optimisation, reduction, and sustainability.
The EN 50600 Information technology. Data centre facilities and infrastructures. General concepts were launched in 2014 with EN 50600:2019 being the latest version of the standard.
EN 50600 is a set of data centre specific design standards that describes the general principles and common aspects of data centre design and operation including terminology, parameters, and reference models. Specific critical infrastructure systems are covered including power, cooling and security, lifetime operations and management including energy efficiency, business risk, operational cost analysis and key performance indicators (KPIs).
For a copy of EN 50600 see: https://shop.bsigroup.com/ProductDetail?pid=000000000030374947
The EN 50600 series was developed by CENELEC, the European Committee for Electrotechnical Standardisation. CENELEC is the European Union’s principle electrical standards authority but the EN 50600 series can be applied globally. This is because the standard has been designed to satisfy the needs of all member countries within the EU rather than at a single nation level.
EN 50600 has developed over time and the International Standards Organisation (ISO) has published several EN 50600 documents as ISO Technical Specifications for Data centre facilities and infrastructures including:
The EN 50600 series of standards therefore provide a solid basis against which to audit a data centre but there are other design principles that can be applied, even to smaller computer and server rooms. The most widely adopted of these is the Uptime Institute’s Tier-rating system.
The Uptime Institute is an advisory body in the data centre industry whose focus is on improving the performance, efficiency, and reliability of data centre facilities through collaboration and performance standards and certifications.
The Uptime Institute is taken to provide the benchmark when it comes to resilience in terms of power and cooling and their ‘Tier Ratings’. Though not a standard, the Tier-rating scheme allows data centres to be audited against specific Tier-levels from I to IV as to their power and cooling level resilience and redundancy levels.
Tier | Delivery Paths | Redundancy | Maintenance | Fault Tolerant |
---|---|---|---|---|
I | 1 | No | No | No |
II | 1 | N+1 | No | No |
III | 1 Active / 1 Passive | N+1 | Concurrent | No |
IV | Multiple | 2N | Concurrent | Yes |
For more information see: “System Availability and Tier-Levels”: /blog/data-centre-design-availability-tier-ratings
The Uptime Institute’s tier rating levels helps to apply a focus on to the critical power and cooling systems within a data centre and on their sustainability and availability in terms of the 9s. A Tier IV data centre offers four-nines (99.995%) availability and is fault-tolerant and concurrently maintainable.
Certification by the Uptime Institute is a multi-stage approach that includes design certification and then
The physical audit is important as actual builds can vary from approved designs and this can lead to a weakness being introduced into physical data centre infrastructure support systems. The TCCF is awarded for a two-year period as technologies change, as can data centre practices.
The Uptime Institute’s tier-rating system is a great way for a data centre to demonstrate both its design and physical building operations in terms of resilience and availability. The same principles can be applied to smaller server rooms and computer rooms when it comes to reviewing power (LV switchboards, UPS systems and PDUs) and cooling (air conditioning) arrangements.
The data centre industry is sometimes quoted as the 5th utility’, alongside electricity, water, gas and telecoms, such is its size and importance to how we work and live. Energy usage and efficiency continues to be a major concern within the industry. Not just the amount of energy used by large hyper-scale facilities but the costs (overheads) for colocation datacentres which must be passed on to clients.
There are several ways to analyse and compare energy usage for both server rooms and data centres.
Power usage effectiveness (PUE) is a ratio developed by the Green Grid to measure energy efficiency in a data centre. As such it is the result of the electrical and mechanical design of the data centre and its implementation during the build-out and operational phases.
PUE = Total Facility Energy / IT Equipment Energy
The reciprocal measure of PUE is Data Center infrastructure Efficiency (DCiE)
DCIE = IT Equipment Power / Total Facility Power x 100%
More info: https://www.thegreengrid.org/
It is important to note that PUE does not factor in sustainability and how the electricity used to power the facility is generated e.g. nuclear or renewable power sources. In addition, PUE does not consider energy reuse. Some larger datacentres may reuse heat created during the cooling process which in turns reduces the total energy consumption of the facility.
PUE is a relatively easy energy metric to compute and can provide a useful benchmark for measuring energy efficiency improvements ‘in-house’ and for comparing to other facilities. In a carbon conscious world, PUE provides a metric that organisations and businesses can use to select data centre colocation suppliers.
The ISO is also evolving standards to look specifically at data centre energy metrics. ISO/IEC 30134 Data centres – Key performance indicators. The standard provides a definition of terms such as Power Usage Effectiveness (PUE) and covers nine KPIs in all including Energy Re-use Factor (ERF) and Water Usage Effectiveness (WUE), and IT efficiency metrics including IT Equipment Utilization for Servers (ITEU_SV).
There are several other ISO standards some of which are specific to the data centre industry which can be included for audit purposes and these include:
ISO 20000-1:2011 can be applied by Cloud data centre providers for Infrastructure as a Service, Software as a Service and Platform as a Service provision.
Further more general ISO standards that can be covered during an audit including the ISO 9000, ISO 27000 and ISO 14000 series:
Most organisations with a substantial investment in their server room and data centre facilities have ISO 9001 as a minimum. ISO 14001 may also be a generic management system alongside OHSAS 18001 or ISO 45001 OHSAS (the replacement standard) for health & safety. ISO 27001 is important from an information security management point of view and may be essential for colocation data centres and as a requirement for suppliers in to cyber security aware industries such as utilities.
For more information see: https://www.iso.org/standards.html
Cyber Essentials is a UK government information assurance scheme operated by the National Cyber Security Centre (NCSC). The certification is the minimum required for organisations to take part in public sector contracts where there is the need to transfer public sector identifiable information. Organisations are assessed against five basic security controls and can be qualified asses for Cyber Essentials Plus certification.
There are other specific standards that can influence a server room or data centre design. These include ASHRAE TC 9.9 2015 covering the ambient operational temperatures for IT equipment. ANSI/BICSI 002-2014 which is US-focused on data center design and implementation best practices and ANSI/ASHRAE Standard 90.4-2019, Energy Standard for Data Centers.
Which design standards to audit a server room or data centre against is often dependent upon the problems or issues faced by the organisation. The Uptime Institute’s Tier-ratings scheme is the easiest as it focuses on resilience and availability, principally in the critical power and cooling systems. Any weakness here could lead to downtime. Calculating PUE and comparing this to other organisations can identify energy efficiency issues that can lead to the need to improve cooling and power provision. An audit to EN 50600 is more comprehensive and can encompass a wider set of subsystems within the facility. Though a longer audit, the outcomes will be a more secure and resilient facility and one that can itself be certified to the ISO data centre standard.