Troy and his team on shift were winding down from a relatively stress-free week and were mere hours away from their normal week off work, when a contract craftsman happened into the control house. “You know the boiler feedwater pump is spraying water into the pipe alley, right?”
As it happened, this was a revelation to the entire crew. None of the instrumentation or associated alarms provided a clue that a serious abnormal condition existed. Serious consequences weren’t limited to reliability. Yes, they had a spare pump, but once the malfunctioning pump was shut down and isolated for evaluation and repair, they were effectively living on the edge—all the “functioning” BFW pumps would now be in service, and the next fault would likely mean critical users would be starved of boiler feedwater. Let alone the fact that, while it was merely water, the temperature leaving the deaerators—close to boiling—was more than hot enough to cause severe injury, and had a record of tragically burning, scalding and killing workers.
In process plants ranging from microbreweries to macro-refineries, hot water in the form of condensate is used to produce steam, provide heating (and cooling), and capture co-produced energy in flue gases, convection sections, incinerators, reformers and reactors. None of these processes operate economically, if at all, without a reliable, quality water supply at a pressure sufficient to overcome and supply steam drums running at 650 psig (45 bar) and up.
In Troy’s refinery, the steam produced turned turbines that not only spun compressors critical to gas-oil hydrocracking and desulfurization, it also powered some of the boiler feedwater pumps themselves and at least one boiler combustion air compressor. That’s why there were four large pumps each capable of hundreds of gallons-per-minute. However, creeping capacity and additional end users made running on any less than three pumps untenable; shorting even a peripheral user like a hydrogen plant of necessary feedwater would topple other dominoes, creating additional complications. Ultimately some big dominos would fall.
Circumstances like this overtake us in the manner of the “boiling frog” urban myth; we wake up in hot water, suddenly realizing it’s a dire emergency. What’s your traditional level of health monitoring for “spared” pumps and motors? In Troy’s case, only routine preventive maintenance and the hopefully keen observations of outside operators on shift provided any clue about impending calamity. Do you hear a new noise today? Is there a drip or a leak where there wasn’t one yesterday? Are your roving operators of a generation challenged to look up from their cell phones?
Large "un-spared" rotating machinery can qualify for a swarm of bearing and vibration sensors, but for spared equipment it’s not as common. If operations gets the notion that they need additional sensors or diagnostics on their BFW pumps to foretell of some impending mechanical breakdown, what would be the likely succession (prioritization) of measurements?
The spray of hot water from the failed pump was attributed to a packing failure, itself caused by a bearing failure. Were the bearings defective from the time of installation and somehow telegraphing their impending failure for days, weeks or months? Or did some subcomponent that had labored for years finally just crack suddenly? Are our machinery PM’s (preventive maintenance) tuned to pinpoint fatal flaws, or are they churned out like the instant oil change you get at the garage?
Alarm management standards and philosophies set a high but clear bar for alarms, that they should be 1) unique, 2) indicative of an abnormal condition, 3) provide clear guidance for a timely operator response that 4) mitigates the likely consequences. We have many alarms that fall short of ticking all these boxes. But in Troy’s case, we have an example that we’d really like to annunciate, e.g., “this important machine is about to break in a bad way.” As we’re seeking technology investments to improve the safety, reliability and situational awareness of our operators, imagine whether “the ideal alarm” can keep us out of hot water.
