Would like to share an article from The DataCenter Journal that talks about diesel generator fuel cleaning and how critical it is in regards to data center maintenance and reducing downtime in your data center - here is the article:
For mission-critical data centers, downtime can be a tremendous expense, and even for less critical data centers and IT facilities, downtime can reduce productivity and sales and cause extensive hassles as personnel scramble to restore operations instead of focusing on normal business operations. A critical part of preventing downtime is implementing adequate backup systems, including backup power sources, such as uninterruptible power supplies and diesel generators. But just having these systems is not enough; they (like the rest of the data center) must be maintained to ensure proper operation when they’re needed.
For diesel backup generators, maintaining the physical machinery is important, but that’s not all. The diesel fuel can become contaminated over time, and using degraded fuel (or even simply leaving it in the generator when not in use) can damage the equipment over time. Ultimately, this can mean that backup generators may not function properly (or at all) when needed. And a utility outage is not when you want to find out about the ill effects of a lack of fuel maintenance.
Bill Miller, regional account manager for Algae-x International, notes, “Fuel for data center generators needs maintenance to ensure optimal quality fuel to the generator at all times. The number one cause of a generator failure is a clogged filter, which is a direct result of fuel degradation that forms tank sediments.” In addition, “The formation of acid in the tank will eat at the guts of the injector pumps as well as the injector tips. The pumps will fail and the injector will no longer aspirate the fuel properly meaning the spray pattern changes allowing for hot and cold spots in the head. A warped head will be a huge headache.” Part of proper maintenance involves checking for water and, if present, removing it from the fuel. One of the threats of water is that it can harbor microbes, which create sediment that can pose a threat to filtration systems. These microbes can and do exist in fuel tank water bottoms. Addition of biocides can also aid in preventing this source of sediment.
But the threat to filtration and other systems in backup generators is not limited to microbial activity: part of the threat comes from the fuel itself. According to Miller, “Biocide dosing can be effective if microbial activity is present, however with or without use of biocides, the fuel will still form sediments as a result of refining processes such as catalytic cracking.” Furthermore, “the vast majority of tank sludge (up to 95%) forms from the fuel itself as the fuel ages. Fuel will break down, form solids and create diesel sludge all on its own, so microbial activity is a minor issue regarding fuel breakdown sediments. This material is what we need to remove when present, and prevent from accumulating by implementing a ‘fuel maintenance’ program.” According to Exxon (“Diesel FAQ”), “If you keep it clean, cool and dry, diesel fuel can be stored 6 months to 1 year without significant quality degradation. Storage for longer periods can be accomplished through use of periodic filtrations and addition of fuel stabilizers and biocides.”
Part of fuel maintenance is inspection of the fuel around the time of delivery. This includes inspecting the tank for water before the fuel is delivered and then inspecting the fuel several days after delivery. “Most commonly, the water is delivered with the fuel. This is the reason for waiting two days after the fuel delivery so the water if any can separate and fall to the tank bottom. The data center is paying for fuel not water,” said Miller. In addition, other steps can be taken to ensure that the fuel, along with the generator, is ready when power failures strike. Miller suggests establishing a tank maintenance schedule “where someone sticks the tank to check for water, pulls a fuel sample from the tank bottom for a lab to test and report and adds a full spectrum catalyst (not biocide) to the fuel every six months and when fresh fuel is added.” And, of course, proper documentation of maintenance procedures and problems is critical to the entire process.
Damage to generator equipment owing to bad fuel is not the only concern. Given the rising cost of diesel fuel, the cost in diesel alone is enough to make regular maintenance important. The longer you can make your fuel last, the less your overall costs will be over time. As in most areas of data center operations, a little time and money put into maintenance can yield much larger returns in the long term, owing to better-performing equipment, less downtime and fewer major equipment repairs and replacements.
Electronic Environments offers diesel generator fuel cleaning and polishing using Algae-X fuel polishing equipment and recommends it as part of your routine data center generator maintenance plan. Do you have your standby generator fuel properly cleaned and maintained?
Then you should know that preventive maintenance strategies are an important part of best practices for any telecom power supply equipment, and a key fact
or in maintaining the 24x7 uptime and availability that is required of any telecommunications facility.
Planned maintenance schedules of critical power supplies optimize uptime for business continuity, as they can often identify an impending equipment failure and prevent it. Regular preventative maintenance of telecom power supply equipment is often scheduled twice a year, although more frequent maintenance may be needed if equipment is frequently subjected to dust and pollen or high temperatures and humidity. Excessive heat and humidity increase wear on equipment, and sometimes can be prevented by improving airflow. Frequent power fluctuations can also have a negative effect on equipment lifespan.
Preventive maintenance for your critical power supplies can extend the life not only of common telecom power components such as UPS and DC power systems, but also of the entire telecom power train, including generators, transfer switches, transformers, and power distribution units (PDUs). Fundamental telecom power components like switches, disconnects, and circuit breakers should also be tested at least annually to verify smooth operation without catching or binding.
As a telecom power supply preventive maintenance contractor, we frequently review the appearance of circuit boards, sub-assemblies and cables, as well as inspect connectors, filters, ups and dc power supply batteries, capacitors and insulation, and check all contacts and connections in the system for looseness, pitting and discoloration. The torque of all connections and the tightness of transistors and PDUs also need to be verified. Dust and humidity can damage switchgear, forming a layer on surfaces that increases component operating temperatures. The dust coat restricts the movement of components, causing excessive wear.
Our comprehensive inspections focus on early detection of problems, as well as immediate replacement of suspicious components.
Electrical thermal imaging of telecom power equipment, especially if it is documented and
tracked over time, can often identify emerging threats for repair and prevent them from actually causing a problem. Thermal imaging (infrared thermography) on electrical connections can help detect overloaded circuits or loose connections which can cause degradation. Abnormal temperature readings for other telecommunication power components like breaker panels, transformers, PDUs or switchgear should also be investigated.
Modern UPS systems have digital signal processor controls that don't suffer from voltage drift the way older units did, but periodically recalibration can still help prevent problems, especially when you replace major telecom power components. Cleaning away dust from the UPS enclosure, and visually inspecting the unit for loose connections, burned insulation or other wear is advisable. Operation of the UPS equipment should be tested at least annually, including battery run-down testing.
Batteries – both DC power plant batteries and UPS system batteries will eventually require replacement. But these critical components should be replaced before they fail, as part of a scheduled inspection program. Replacement will be necessary sooner in elevated temperatures and where batteries frequently cycle due to poor utility-provided electricity. Electrical resistance may increase at the terminals of a deteriorating battery, generating heat and accelerating battery failure unless it is noticed and the battery is replaced. With proper measurements (battery trending analysis) and a good visual inspection, a weak cell can be targeted before it turns to a dangerous condition. UPS and DC power plant battery preventive maintenance can provide detailed information regarding the complete battery plant. Take a look at our blog post on maintaining and extending the life of your UPS batteries.
The take away from this post – telecom power supply preventive maintenance is a key factor in maintaining continuous network availability in your communications facility. Many component and equipment failures can be prevented by steps as simple as dust mitigation and scheduled inspections and as-needed filter or belt replacement - however, a comprehensive preventive maintenance routine is recommended .
What are your thoughts on the best maintenance practices of your critical power equipment? Are there other telecom power supply maintenance issues you see in your facility?
Are you familiar with DataCenter Dynamics? They offer everything to help manage your data center from their premier Data Center Conferences to their variety of training courses and their Focus Magazine. They live up to their description as "being a full service B2B information provider at the core of which is a unique series of events tailored specifically to deliver enhanced knowledge and networking opportunities to professionals that design, build and operate data centers.
Over 16,000 serior data c
enter professionals attended a Datacenter Dynamics event last year. These conferences pull together the people, processes and technology necessary to execute a world class data center - all under one roof. No matter what a your role is in the data center value chain, you will benefit from the insight of the world’s industry-leading practitioners presenting case studies or technical papers on how to optimize internal and external IT requirements from a facility, IT, and business perspective.
Electronic Environments will be exhibiting at the upcoming DatacenterDynamics Converged conference in New York on March 13th. Stop by our booth and say hello - see a HotSpotr air mover in action -- these thermostatically controlled air movers remove data center hot spots immediately and efficiently. Talk to us about our other data center services and solutions as well and register to win a Kindle Fire.
If you can't make one of their conferences - then check out their web site where you can create an account and get acess to great content such as the FOCUS magazine, their blog and much more - www.datacenterdynamics.com
Data center construction is one of the main services offered by Electronic Environments Corporation. Recently Construction Executive Magazine interviewed Mike Kingsley, President and Jim Stark, Design/Build Manager of Electronic Environments for an article based on data center construction and data center power and energy savings.
The article, featured in the February issue of the magazine, titled: "Data Center Clients Challenge Contractors to Deliver More Power While Using Less Energy" discusses the challenges of data center construction today. It discusses issues such as how the economy has changed what IT professionals are planning for their data center needs, such as upgrades vs. new builds. Another common challenge is the fact that the physical foot print of data centers is getting smaller yet the requirements to power the servers and IT equipment is increasing.
The article goes on to discuss the challenges of data center cooling and the best practices and solutions that are available for data centers that are faced with the need to improve their server room airflow and improve energy efficiency within existing facilities as well as ways to increase the building’s energy efficiency and reduce its environmental impact when building a new data center.
Take a look at the article and let us know your thoughts on the challenges you are facing in your data center.
A study of factors affecting data center airflow in a closed aisle.
Written by Rajesh Nair, Founder & CTO of AdaptivCool, an EEC partner
What happens to the data center airflow through a high-density server when the cold aisle it is in is capped or contained?
Of course, no one would let us do this experiment in their critical data center. So we had to resort to simulating the effect on a computer using a CFD tool.
Every server in a rack is designed to pull enough air through it to cool the electronics in it under all specified environmental conditions. The front and rear doors of a rack and cabling in it can add additional series flow impedance to the internal impedance of the server. The server fans can overcome additional impedance to some extent since they are normally over-designed. Having enough cold air available in an open cold aisle helps overcome some of these airflow restrictions. However, when the cold aisle is capped (contained) the situation changes a lot.
In the simulation study below, we modeled an existing data center. The pressure pattern along the whole data center cold aisle is shown in Figure 1 as shaded plane. The pressure stays relatively flat along the whole length of the room. We modified the model by capping the cold aisle and studied the pressure in the same plane. Nothing else in the room was changed. As the figure shows, the pressure in the cold aisle dropped significantly. A lower server intake pressure increases the pressure difference the server fan has to work against, causing the airflow to drop. Here is an electrical engineer’s analysis of aisle capping on server pressure and airflow.
Airflow through a server enclosure:
Server boxes are designed to operate under wide thermal environmental conditions. The built-in fan is capable of pulling enough air to cool the electronics in the box. In Figure 2, a simple electrical representation of airflow is shown. The air intake and exhaust are the open room space and are at the same pressure. The fan creates enough pressure (Ps) to create airflow (Q) over the internal impedance (Zs). The fan speed determines the driving pressure and airflow. The internal control system modulates the fan speed to draw just enough air to cool the electronics. Once this server is placed in a rack it sees additional impedance, Zr, from the front and rear doors and cabling in the path of airflow (Figure 3). The server fan now needs to operate at a higher speed to compensate for these series impedances to get enough airflow to cool the devices inside. This may be acceptable to a limit since these systems are normally over-designed. But at the extreme thermal conditions this additional impedance, Zr, may limit the cooling capacity of these servers. So it is important to minimize Zr.
Airflow in a closed aisle:
Once the cold aisle is closed, the airflow-pressure scenario changes further. Since the aisle is closed, the aisle pressure (Pa) may not be at the same as the pressure in the hot aisle. As shown in Figure 4, there are two series airmovers in the airflow path; the CRAC and the server fan. The pressure created by the CRAC (Pc) causes airflow under the floor to overcome the under floor impedance (Zu) and the impedance of tiles in the cold aisle (Zt). Depending on the distance from the CRAC to the aisle and underfloor impedance, the airflow could vary significantly from one closed aisle to another. In Figure 1 the CRAC fans were not creating enough pressure to discharge air into the closed aisle. In fact, the fans in the servers were pulling cold air into the aisle thus causing a lower pressure in the aisle than the rest of the room. The net airflow through the servers dropped significantly once the aisle is closed. One fix to increase airflow through the servers is to increase the aisle pressure by using more open grates (and thus reduce Zt). This could still only be a temporary solution. The pressure in an aisle is influenced by multiple CRACs. If the CRAC with highest influence is turned off, the aisle pressure can immediately drop and other CRACs may not be able to compensate.
Pressure compensation in a closed aisle:
The way to guarantee airflow through servers in a contained aisle is by controlling the pressure in the aisle. An effective way is by use of either a fan assisted active tile (Demand Based Cooling system from AdaptivCool), or fan assisted chimney or rack backdoor with automatic speed control. The last two options need to be installed on each rack, as these manage pressure drop only within a rack. The active tiles (Figure 5) can control pressure in the whole aisle and compensate for CRAC failure. The cold to hot aisle pressure difference is used for fan speed control. This approach can offer normal server operation in a closed aisle even under CRAC failure conditions. Figure 6 is a representation of airflow and pressure in the room.
Closing:
The primary advantage of data center aisle containment is improved cooling efficiency, by avoiding mixing of cold and hot air in a data center. It also comes with unintended consequences. The two important ones are decreased data center airflow and loss of redundancy to CRAC failures. Both of these can be addressed by using additional fan assisted tiles in the airflow path that can hold a steady aisle pressure as well as compensate for CRAC failures.
To learn more about data center airflow best practices, fill out our contact us form or contact Brad Morgan, our Data Center Cooling Solutions Manager at bmorgan@eecnet.com or 508-229-1446.
During extraordinary events such as the Santa Ana Wind Storm in Southern California, Hurricane Irene and the rare October Snowstorm that hit the Northeast, EEC requires all personnel to be on-call and available to handle multiple emergencies for our critical data center and telecom customers. Take a look at what one of our data center customers had to say about our response during Hurricane Irene:
Access Northeast is a rapidly growing Data Center Provider with a focus on Colocation, Cloud, Hosting and WAN Management. We provide 24x7x365 mission critical services to our clients with an expectation of 100% application availability. As part of our emergency preparedness procedures in advance of Hurricane Irene in August 2011, we deployed additional staffing to our Marlborough, MA and Waterbury, CT data centers to monitor these facilities during the storm.
At approximately 1:00AM on Sunday, August 28, 2011 during the full brunt of Hurricane Irene, our Waterbury Data Center Manager, Douglas Barry, identified a leak at the condensate pump on one of our air handling units. He immediately called the Electronic Environments Corporation (EEC) dispatch center to request an emergency site visit. At 2:30 AM, EEC technician Steven Thibodeau arrived onsite. Steven was able to clear a blocked drain and returned the unit to full operation within the hour, thereby preventing any service impacting issues from transpiring that could have affected customer uptime.
Responding to an emergency maintenance call in the middle of the night is no easy task. Driving through the heavy rain and dangerous winds to reach a service site is also daunting. Resolving an HVAC unit issue in a timely manner is a tremendous benefit to any service provider. However, delivering all three in unison is the EXACT reason why Access Northeast has chosen EEC as our facility maintenance partner throughout New England. Due to the extraordinary efforts of EEC and Steven Thibodeau, Access Northeast was able to deliver uninterrupted service to our customers throughout the duration of Hurricane Irene, unlike many of our competitors in the region. Special thanks to all parties once again.
Sincerely,
Michael Donnellan,
Access Northeast Chief Operating Officer
Don't get left in the dark the next time there is a storm. EEC's 24x7 e
mergency repair service program is an aggressive and rapid response mechanism developed specifically for Data Center and Telecommunication operations. Our philosophy is to aggressively and rapidly respond to instances of system failure with the right personnel and the right material to bring your site back up immediately.
Our technicians arrive with "crash kits" containing replacement parts for 90% of failures and have access to a vast parts inventory - we will do whatever it takes to get your systems back on-line fast.
Join Electronic Environments and Packet Power for a free webinar:
Data Center Power Monitoring Made Easy
Recorded version now available > >
Power is an essential part of data center operations. Data centers must make sure enough power is always available. This has always been a challenge, but the challenge has grown as data center operators have to contend with delivering far more power per square foot of space -- and delivering it more efficiently.
Join us and you will walk away with great tips on how power monitoring can help you:
- Improve operational effectiveness
- Better allocate power costs
- Extend the life of your data center
You will also get:
- A comparison of data center power monitoring options
- Benefits of wireless power monitoring
- Simple ways to use monitoring data
Recorder version of webinar now available - click here > >
For more information on Electronic Environments or Packet Power and the soltuions and service we provide, please go to:
www.eecnet.com www.packetpower.com
The easiest, most flexible and cost-effective way to monitor power in your data center.
I just saw an interesting article on Computerworld.com "A tale of two U.S. government data center projects" where one project came in on-time and within budget and one went over budget and was late. The moral from the story is that the project that went over budget used the Design-Bid-Build (traditional bid/spec) approach and the one that came in on-time and within budget opted for a design/build approach – where you have a single point of contact and that contact is very familiar with the project from the start and can make any necessary changes fairly quick. Many times, as in the case of the over budget project, when a problem arises you have to negotiate with multiple teams to get things fixed which more often than not results in delays.
At EEC we work with our clients on either type of approach, however, we recommend the Design/Build approach. The need to bring technology projects on-line fast challenges traditional construction practices. The drawing below depicts the scheduling difference between a traditional construction schedule and a design/build schedule on a typical job:
The typical characteristics of the design/build approach are faster completion time (as represented here), the potential to be more cost effective, and turnkey responsibility is low risk for the client.
The data center construction phase is impacted by all the preceding events, missing even the smallest detail could wreak havoc later on. EEC has project managers 
that are thorough in making sure that all key milestones are being followed and are on track. The procurement phase is a critical component of construction management; our project manager devotes considerable effort and attention to the organization of this function. They establish key dates and monitor submittal and delivery cycles for long-lead equipment. This requires a keen eye on vendor management – developing an early, clear, and concise view of every vendor’s role.
Also, our internal network of resources including our product specialist, electrical and mechanical engineers, and specialized technicians all work together to make sure construction standards, schedule, and design specs are adhered too.
Have you recently built a data center or planning to build one? Which approach did/will you take? We’d like to hear from you.
There are a few important questions that facility and data center managers need to ask themselves to make sure they are on the right track in terms of timely and proper com
puter room air conditioning (CRAC) equipment maintenance; these include: What maintenance needs to be done on the equipment? When? How often?
Preventive data center maintenance is the best approach a company can take to limit the amount of downtime on equipment in critical environments. Within the CRAC system the most common failures tend to be high temperature or high head failures - both of which can be prevented by changing air filters frequently as well as by power washing the condenser to maximizer air flow and to maintain the operation of the unit.
I recently came across an article on SearchDataCenter.com, "Best practices for maintaining a data center CRAC unit". This article reviews several best practices that are important for maintaining your data center CRAC unit. Here are some of the items it covers:
-
Cooling is too critical to leave to chance
-
Fear of intentional cooling shutdown
-
What CRAC unit maintenance should entail
-
Taking care of mechanical items
-
The importance of refrigerant levels, electrical testing
-
Making time for external maintenance
Be sure to take a look at the article, it offers some great information in regards to maintaining your Computer Room Air Conditioning units.
And don't forget - performing maintenance can help you save energy
Perfo
rming maintenance can also help equipment run more efficiently, which in turn helps with the longevity of the equipment, but more importantly saves money by reducing the amount of energy the equipment uses. Scopes of work to help equipment run more efficiently should include regular checks of the quality of fluids, frequently changing air filters, and power washing condenser coils to name a few. Checking fluids is essential because contaminants can result in diminished performance. Changing filters frequently is important because it will maximize the airflow in the equipment. One of the most important maintenance checks on CRAC equipment is regularly cleaning the condenser coils. Condensers are known to draw in dirt, pollen, and other debris that restricts airflow and reduces efficiencies. According to the Consortium of Energy Efficiency, facilities in which proper computer room air conditioning preventive maintenance is completed will use approximately 15% to 20% less energy than those where systems are allowed to deteriorate. With the high cost of energy it is crucial to maintain the equipment in a data center in order to take advantage of efficiencies and save money.
Companies should work with their maintenance provider to develop a scope of work - with the exorbitant cost of downtime, a strong preventive maintenance program is not an option - it's a must!
Dirty fuel is a fact of life. Even with the development of cleaner burning fuels, contaminants are still a major concern when it comes to fuel systems. Micro-organisms can quickly become a major problem. Colonies of fungus and bacteria feed on your fuel.
Commonly known as algae, they spread rapidly in the presence of moisture. They easily spread from one tank to another by riding in the fuel. As these micro-organisms accumulate, they will spread through your data center's generator fuel system and quickly plug the fuel filter. Most end users will simply add a biocide to kill the fungus. While this does kill the bacteria, it does not remove the growth media from the tank. Biocides are also very toxic and cause chemical burns when they come in contact with the human skin.
The correct solution to the problem is to remove the water. This can be done by polishing the fuel in your generator fuel tank. Water is removed by three common methods:
- Stripping-which is a silicon based medium that inhibits the passage of water but allows fuel to flow.
- Coalescing-Gravity drags water droplets out of the slow-flowing fuel.
- Absorption-A filter medium with a high affinity for water and a low affinity for fuel absorbs the water.
- Algae-X in-line conditioner - by stabilizing and conditioning the fuel, reducing the size and mass of clusters of fuel molecules.
In addition to the above methods, particulate filtration is defined as passing a fluid through a filter media that has a specified micron rating. The fuel is allowed to pass and the particulate matter will be caught in the filter media. Bear in mind that the particulate matter that is smaller than the micron rating of the filter media will also be allowed to pass through the filter. Because of this, it is important that your primary filter be of a higher micron rating for the primary fuel pump's protection and that your secondary filter be of a smaller micron rating to protect the fuel injection pump and injectors.
What is your plan for keeping your data center up and running 24x7? Make sure fuel cleaning/polishing is in that plan. To learn more contact us.