Keep Hot Miners Cool


Background

Cryptocurrency mining has a large impending roadblock quickly approaching. If it is not addressed and properly dealt with, home mining will be impossible, and even commercial mining will be severely impacted.

Heat is an inevitable byproduct of the use of specialized computers for the generation of cryptocurrency. A single mining machine has as much heat energy as a large full grown steer. So, a 15″ long electrically powered device has to be viewed like a Texas Longhorn for the purpose of heat management in order to understand the problem. A small mine operator might have a few dozen devices setup to harvest cryptocurrency, essentially a herd sometimes crammed inside a closet. 

Two dozen currency miners.

 

Mining Machines Turn Electricity Into Heat

A common mining machine can use 1200 watts 24 hours a day, 7 days a week. Mining power supplies are designed to handle constant power draw and incredibly high amperage. The Bitmain APW5 can easily handle over 100 amps constant output. The mining machine in this example would generate over 4400 BTUs of energy, with capacity of double with just this power supply, or nearly two full grown  steers.
Managing heat for miners means managing the heat energy equivalent to the heat energy from a 2000lb animal for each mining device.

Equipment Cooling Options

Cooling has moved through a few significant milestones as computing technology has put more energy in a smaller place. Mining hardware is directly comparable in computing power to some of the highest end high performance, multi-CPU servers. Enterprise computing is running into the thermal management problem. Faster, more capable computers are able to consume more electricity and convert it to work with a byproduct of heat. The roadblocks in all computing becomes heat management, light speed, and the ability to perform more work for the same power while also being able to shed the heat quicker. With mining devices, significant innovations have been directly associated with the ability to perform more work for the same power. Unfortunately, thermal management has been limited to attaching fans, larger fans, more large fans, and fans that spin so fast that they sound like helicopters.

An overview of cooling strategies and available technologies will be discussed herein.

Air Cooling

Air is an inefficient medium for heat transfer. In a snowy winter, you would be safe outdoors with a coat and gloves. Now, fall through ice into a 35 degree (0 C ) lake. The heat loss would be almost instant with clod fluid, compared to a blowing winter storm. 
If we are installing the herd of steer collection of miners, we create challenges because of practicality and reasonable limits. First, we don’t have a large field within which to deploy two dozen miners, nor is it reasonable or convenient to run power across a field to support the miners. While it might help to cool them, a rain storm would likely destroy the devices permanently. Miners are installed in a condensed area. This makes is practical to supply power to the area, racks for mounting the devices and cooling which we will explore.

Inlet Air Supply

A single miner, like an antMiner S9, uses two high speed fans to cool each one. Fans are typically rated by cfm (cubic feet per minute) of air moved. While background and research produce number between 200 – 230cfm at 6000rpm, we will use the higher number. Each miner has two fans. Bitmain configures the rear fan to spin faster than the front fan. The idea is that this creates a slight differential. Since the fans are in series, only the rear of the set matters, so each pair of fans on a miner produce a net 230cfm of air movement. With 24 miners, that is 24 x 230, or 5520 cfm.
The outlet air from the miner is moving at 5520 cfm, and is over 100 degrees F (38C).
The inlet air is moving at the same speed, less the reduction in speed from the inlet fan and any inefficiencies – like drag within the miner itself. Allowing -10% for the front fans of 24 miners, the miners need 4968 cfm of air delivered at a cool temperature. The chips run at 170F (75C) approximately, so inlet air of 60F (16C) will allow internal chips to shed excess heat. In a 15″ length, the air enters at 60 degrees and increases by more than 110 degrees. Draw inlet air from the floor in the server room, or use tubing channeled from the floor to the miners.
If the inlet air is hotter going in, the capacity of the air to absorb heat is reduced. If the air is 170 degrees going in, it can’t absorb heat at at rate that the chips will stay at 170, so the hardware will heat up quickly. This happens in areas with insufficient air flow. The inlet air starts to get hot, since it is warmed by the outlet air. While the inlet air is unlikely to get to 170 before the machine shuts off, it can easily exceed 100 degrees. This creates a problem since the machine has 15″ to shed 110 degrees. If the air is already 100 degrees, it may only be able to absorb a fraction of the heat. If the outlet temperature raises to 120 degrees, this means that the miner only shed 20 degrees, but needs to shed 110 degrees to maintain an operating temperature of 170 degrees. The remaining 90 degrees warms the miner, raises chip temperatures, and requires more air to cool. At a certain point, it will end up feeding so much heat into the miner that it can no longer shed heat. The miner will either fail from heat, get damaged or protectively shut down.
Understanding this, to manage heat from these miners, they need to be fed fresh, cool, dry air with 4968cfm at the inlet fan. If the miner has an inlet hose, that hose will create drag. It will slightly lower the available air to the miner, and will lower the air density. To deal with this pressure differential in front of the fan, 4968cfm behind the fan will be 10% greater for every ten feet of away from the miner the air inlet source is. If the air is being pulled from a constantly pressurized air conditioning zone, 12ft away, it will require 4968 x 120%, or 5962cfm, rounded to 6000cfm at the AC, constantly delivered. If it is an external source, the inlet fan drawing air from the outside has to deliver 6000cfm. The inlet source (AC or a vent from outside) should be contained in a conduit at least t he same diameter as the inlet fan. If this is not possible, add a fan booster for every 10ft. This will maintain a constant inlet air line pressure sufficient to feed and cool the mining devices. The miners will be fed 6000cfm inlet air into a sealed room, from which the inlet fans will draw 4968cfm of air to push through the miners. Sealed tubing should be used from the inlet air source to the miner room, ideally directly into inlet horns added to the miners.

Eliminating Outlet Air

The outlet has to be planned with the same care to keep the miners, the electronic herd of steer, from overheating. The miners combined are exhausting 5520cfm of air heated to 110 degrees. This air has to be vented ideally 50ft from the inlet source (if both are outside) and seperated by at least 90 degrees and one structural corner. In other words, outlet air and inlet air should not share a common external wall. Capture the outlet air with 6 inch venturi attachment. Onto that, attach galvanized dryer hose. Use formed elbow on galvanized pipe to move hot exhaust from back of miners up towards ceiling.
Each miner will have a galvanized pipe turning the outlet air flow from horizontal to vertical, then up to the ceiling. On the ceiling, a galvanized steel manifold can allow all the pipes to converge. For 24 miners, this will likely be 12″ x 24″ x 72″. In this case, there would be 4 miners connected to the manifold per 12″. One one side of the manifold, an electric thermostatically controlled fan needs to pull the air through the pipes, through the manifold and then to the outlet. One end of the manifold is opened up, fitted with two 12″ flanges, and two 12″ line boosters. The combined capacity of the two fans needs to exceed 5520 x 120%, or 6624, rounded to 7000cfm. Each 12″ line booster needs to generate 3500cfm or more.
Two 12″ galvanized steel lines are added to the fans, routed to an external outlet location. Most buildings have integrated ventilation in their top floor, in an attic area. The hot air can be ventilated to this vented area – if a full time or thermal outlet fan exists. That fan is sized using the same formula as the intake – fan booster per 10ft distance from outlet – at back of miners. Assuming that the air travels 20ft from back of the miner, then up a floor, then across to an attic area, this will require an additional pair of 12″ 3500cfm line boosters, and (if released into an attic space) the attic fan has to be sized to handle 7000cfm more than the normal load. In hot areas, this may require an additional fan. In cool areas where the fan is on only once in a while, 7000cfm fan capacity is sufficient.

Air cooling 24 miners requires moving 6000cfm of fresh, dry air into the miners, at a cool temperature, then getting rid of 7000cfm of hot air, moving it far away from the  miners quickly.

Liquid Cooling

Liquid cooling has moved through a few significant milestones. I will highlight a few.
 

Water Cooling

Liquid cooling (aka water cooling) uses thermal transfer blocks, tubes and radiators. Water that is carefully distilled can be nonconductive, but this is a stretch, and not likely. As a result, water is stored in a reservoir. It is cool in storage. Pumps draw it from storage to tubes directing it to heat transfer blocks mounted on processors – CPUs, GPUs, ASIC hashing chips. The water is used as the thermal exchange medium, carrying the heat away from the thermal block. The water goes to a radiator or a chemical cooler. From there, back to the reservoir, and the cycle repeats. Water is electrically conductive and minerals within the water can damage electronic components. Bad fittings, leaking joints, assembly issues, heat and time can degrade the performance and reliability of a water cooling system. In practice, water cooling systems use glycol based (or similar) coolants, with UV reactive dyes. The coolants resist the buildup of deposits, have superior thermal transfer performance than regular water. Water cooled systems are dry when disassembled, but thermal transfer blocks need to be removed and reattached properly.

Oil Cooling

Mineral oil is a dieletric fluid. This means that it is not inherently electrically conductive, unlike water. Mineral oil is an excellent thermal transfer medium with a very high boiling point over 300 F. The high boiling point means that a mineral oil based system needs to be circulated and cooled, then added back to the system, like water cooling. The difference is that since it is not electrically conductive, all the electronics can be fully submerged, rather than the fluid needing to be contained within metal and tubing channels. The full submersion allows better thermal management and transfer. A mineral oil system depends on convection circulation and pumps to circulate the fluid out of a central container, like a tank, into a radiator, and back into the tank. With mineral oil cooling, the tank acts as the reservoir and the thermal transfer area at once. The pumps draw the hot oil from the top of the tank, move it to an external radiator (or cooling system) and introduce it to the bottom of the tank. The problem that mineral oil has is that it is messy. It is a light oil, but parts in the oil will come out oily. Mineral oil is corrosive to certain plastics and rubber material. Things like the plastic for the PCIE slot on a motherboard might get damaged, along with tubing, fittings and accessories.
 

Immersion Cooling

Modern 2 phase cooling fluids work exactly like the oil as a heat transfer medium with one important different. Instead of being mechanically pumped through radiators or chemical coolers, they use evaporation to lose stored heat. 2 phase cooling fluids have lower boiling points, 45 degrees C as example. The cooling fluid boils. The boiling moves the heat out of the fluid into an air space above the fluid. The fluid condenses within the chamber, transfers heat to metal or liquid in copper tubing. The cooling fluid condenses, then falls back into the container. In a 2 phase system, it can be built like a mineral oil tank system, but without the pumps, external radiators and the need to pump the fluid in and out. When the fluid cools, convection lets the fluid fall to the bottom of the tank, and raises as heat is transferred to it, until it boils, and the process starts again. Electronic parts removed from 2 phase cooling fluids come out dry, with no residue.
 

Hybrid Immersion Cooling

The 2 phase system is pumped through lines and heat transfer blocks, much like water cooling. The system is sealed and pressurized. The pressure allows raising the boiling point of the fluid. It circulates through the heat transfer blocks and contained areas under pressure. Pumps circulate it through the system from high pressure areas to low pressure areas. In the high pressure areas, it collects heat. In the low pressure areas, it rapidly dumps the heat as it boils, and is then chilled to lower than atmospheric temperatures using cryogenics, refrigeration or a combination of both. In this way, the thermal transfer fluid can lower the operating temperature significantly while the system runs with otherwise aggressive overclocking. With a hybrid system, thermal transfer blocks need to be properly disassembled. A neat benefit of the hybrid system over an open 2 phase system or a water cooled system is that the hybrid offers performance and efficiency over the two other systems. The hybrid system uses pumps to circulate the fluid. There is a greater heat exchange at lower fluid movement speed. As a result, the hybrid system uses much less power to cool than conventional water cooling or air fans. In comparison to the standard 2 phase Immersion cooling system, or an open system versus a closed system, the closed system can achieve equivalent or superior cooling performance with less
fluid. The open system needs a large volume of fluid to capture heat at a such a rate that the excess heat is managed. The hybrid, or closed system, creates more surface area to which heat can be transferred to quickly, by moving cool fluid past the warm areas. The failure risk is higher in a pressurized system. Every joint, every line, every seal is subject to higher operating pressure. In the areas where open tanks are used, parts can be removed and added without concern, but all the seals must be verified to maintain the performance of a pressurized system.
 
2 phase fluid is expensive, around $400/gallon. Standard 2 phase systems use larger tanks, consuming larger amounts of cooling. Hybrid systems use less fluid, but require more accessories. The cost of liquid cooling is offset by the longevity and performance of the cooled systems. With more advanced cooling, systems can run with otherwise aggressive overclocking, yet operate at lower temperature and reduced power draw. Cooling allowances must be made for all mining systems. At a certain performance level, the cooling system can pay for itself in place of the air cooled alternatives.

Summary

Cooling is an often overlooked, but incredibly important part of cryptocurrency mining. A failure to plan thermal management will result in rapid, excessive heat accumulation and hardware failure. We are building modular 2 phase and hybrid cooling kits for ASICs. Our development has been to increase longevity and performance of our own hardware, but we are open to offer these to others if there is sufficient interest.

 

About SafeView

SafeView is a brand developed and owned by Airius Internet Solutions, LLC. Airius has been in business since 1999, helping regional and enterprise global organizations with hardware, software and solutions strategy. Following the economic shifts that happened in the early 2000s, Airius worked closely with business and government to apply innovative ways to look at unstructured risk and vulnerability data. Solutions were developed for government, global financial and innovative technology companies in the years that followed. The worldwide financial crisis that followed the US based banking collapse stimulated the  creative impulse that became cryptocurrency. Airius has advised global financial organizations regarding the  use and incorporation of blockchain technologies for assurance and decentralized consensus of transactions. With the growth in commercial mining, Airius engineers developed SafeView Mining Manager, a web based mining business management dashboard, customized air and liquid cooling solutions and show quality mining machines on contract to vendors, manufacturers and fortunate customers. Airius publishes tools, utilities and guidelines for mining here and on Facebook.

Please contact Airius with your questions at info@airius.com.

  • Ernest Park, CTO and Editor, SafeView

Comments

comments