Solving Riboflavin Failures with First Principles: A Tank FAT Case Study

The Crisis: Project Schedule Slippage

This case study illustrates project slippage due to a tank Factory Acceptance Testing (FAT) failure. For project managers, this case study is a lesson in trusting data over position.

A project manager for a fast-paced design-build life sciences project was facing imminent project slip after a critical tank repeatedly failed FAT testing over the course of two weeks at the tank fabrication shop.

The Challenge: The Persistent Riboflavin Test Failure

The failure occurred during the tank wetting testing. Wetting is used to show that the tank surfaces can be contacted by water using a spray ball. Wetting represents the way a cleaning solution can be expected to wet the tank during cleaning in place (CIP). For wetting testing, a small amount of riboflavin is dissolved in water and then sprayed onto the internal surfaces of the tank. Wet riboflavin fluoresces green when exposed to ultraviolet light. The tank wetting test is completed by delivering water through the spray ball(s) at a known flow and pressure. The tank internals are inspected while wet for any fluorescent liquid remaining. Generally, testing is successful after a few parameter adjustments such as additional holes are added to the spray ball or water flow/pressure are adjusted.

Management Directives: “Beating the Dead Horse”

The project included a geographically dispersed team with a project manager in Ohio, a fabrication shop in the Northeast, and a senior management team in California.

Each time the tank failed the riboflavin testing, the inspector accurately reported there were traces of riboflavin remaining in the tank. After several early test failures, the team sought guidance from their senior management. Management instructed the team to insist that the tank fabricator provide a correctly drilled spray ball and repeat testing until the fabricator provided the correct spray patterns to pass the test. Several spray balls with different hole configurations with an ever-increasing number of holes were tested.

Calling in the Cavalry: Genesis’ Senior Process Engineer Arrives

Under pressure to meet milestones, the project team faced a scheduling disaster if the delayed tank didn’t ship. The blog author, Genesis’ Senior Process Engineer, was directed to visit the manufacturer and observe the FAT and determine if the tank could be delivered in a timely manner.

Upon arrival at the fabrication shop in the early afternoon, another riboflavin test was in progress. After a failed morning test, adjustments were made, and the afternoon test was almost complete. The tank inspector exited the tank and announced that substantial amounts of riboflavin remained on the tank walls. The FAT team was disgruntled with the test results and they wanted to communicate with their management.

Before the team adjourned, the author suggested that the spray ball was not the problem. The team was aghast. They were taking direction from vice presidents at the company headquarters who they stated “surely knew far more about FAT testing than anyone else in the fabricator’s facility.” As the FAT team sashayed to the conference room for their call, the author began an investigation with the tank inspector.

Using First Principles: The Scientific Method in Action

Having already formed a hypothesis that the spray ball was not the cause of the test failures, the author asked the inspector to describe the pattern of the residual riboflavin he saw in both the morning and afternoon testing. Next, he asked for confirmation that this was the same pattern he had observed during previous testing. He confirmed all the patterns were similar with noteworthy streaks and that while the streaks and the concentration of the riboflavin patterns varied slightly, they were still observable in every test.

The Diagnostic Method: Ruling Out the Spray Ball

Armed with this information, the author collaborated with the president of the fabrication facility to create a field test to rule out the spray ball as the cause of the failures. With the help of his shop foreman, four three inches by three inches stainless coupons were cut from the same stock as that of the tank. Half of each coupon was treated and passivated as if it were a tank surface. The tank wash solution was applied to the four halves. Three of the halves then experienced one, two, and three water rinses, respectively. When dried, all four coupons had their soiled sections sprayed with the same riboflavin solution used in the tank. After the riboflavin solution was tacky to the touch, all coupons were misted using a spray bottle with tap water until thoroughly wet to mimic the effects of cascading water over the internal tank surface.

The ultraviolet lamp used to detect residual riboflavin was illuminated over the coupons. All the coupons evidenced riboflavin streaking. Each coupon displayed decreasing concentrations of the streaking that aligned with the greater amounts of rinsing applied with the misting bottle. The streaking patterns observed were confirmed by the tank inspector to reflect those observed in the tank.

Based on this discovery, the tank fabrication shop immediately sent the tank for additional water rinsing prior to testing the next day. The FAT team’s meeting shared the information with senior management who demanded that no further testing be done until more holes are added to the most recent spray ball.

The following day the tank passed the riboflavin test on the first try. The fabrication shop permanently changed their post passivation tank washing protocol to ensure there would be no further riboflavin streaking due to surfaces containing trace detergents on any future tanks. The FAT team took full credit for the successful test attributed to the new spray ball hole pattern. The tank shipped in time to meet schedule.

The Conclusion: Detergent Residue Was the Culprit

Multiple spray ball configurations with varying number of holes failed to remove residual riboflavin from the internal surface of the tank because spray coverage was not the cause of the residual riboflavin. After passivation with acid, the tank internals were washed with a detergent to remove particulate and any remaining passivation solution. Washing was followed by rinsing to remove micelles and detergent residue. Coupon test data confirmed that the more rinse water was applied to the coupons, the less riboflavin streaking was observed. The logical finding is that the tank internals were contaminated prior to being sprayed with the riboflavin test solution. Removing the contamination, trace detergent with surfactant, allowed for a clean surface to apply the riboflavin test solution, and its subsequent easy and complete removal with water from a spray ball.

Key Takeaway: Lessons for Your FAT Team

Ensure that the FAT team consists of at least one person with sufficient experience and understanding of the testing process to facilitate issue resolution in the event of an upset condition. Rely upon real and reproducible data when evaluating an upset condition over that of conjecture or position.