2013: MLM module is hit with a contamination disaster
In 2013, when the MLM/Nauka module was believed to be less than a year away from launch, engineers testing its systems made a startling discovery. The spacecraft's critical propulsion system turned out to be heavily contaminated with metallic dust. It was spread around the maze of pipelines, valves and combustion chambers during the "upgrades" of the spacecraft in the previous years. One almost unbelievable rumor had it that workers charged with sawing off redundant components thought they had been cutting the vehicle to scrap! Here is the inside story of the MLM module contamination blunder.
Engineers at GKNPTs Khrunichev work on the MLM module circa 2013.
From bad to worse
At the beginning of 2013, the Russian space agency, Roskosmos, held good hopes for restarting the long-stalled assembly of the International Space Station, with the launch of the 20-ton Multi-purpose Laboratory Module, MLM, dubbed "Nauka," Russian for "science."
On December 14, 2012, following a trip from its assembly plant at GKNPTs Khrunichev in Moscow to the nearby town of Korolev, the MLM/Nauka module was ready for a comprehensive series of tests at RKK Energia's Control and Checkout Station, KIS. It would be the same place and the same procedure that most Russian manned spacecraft have undergone over the years, before being delivered to Baikonur Cosmodrome for launch.
From April 2 to 4, 2013, six primary and backup crew representing three upcoming long-duration expeditions to the ISS conducted training inside the flight-worthy module.
If everything went as scheduled, the MLM could reach the launch pad at the end of 2013 or beginning of 2014... But, things did not go as scheduled at all.
On May 28, 2013, during functionality tests of the MLM's propulsion system, engineers found a leak in the so-called EKOD2 valve. That particular component had the designation DEGK 17KS.60Yu6003A-0, Position 588. Designed for high-precision performance, valves of this sort are extremely sensitive to any contamination. To the surprise of testing specialists, the problem was traced to metallic particles in the internal cavity of the valve and in the adjacent pipelines.
The damage was serious enough to require a complex procedure of cutting away the valve and welding in a new one. Before committing to the repairs, engineers had to practice the operation on a full-scale prototype of the MLM module known in Russian as Kompleksny Stend, KS.
What was much worse, this was not an isolated incident. Further inspections of the module at RKK Energia found a widespread contamination inside the propulsion system, which would require either cleaning or the replacement of affected components. According to very preliminary estimates, it would take up to 10 months to resolve all the contamination issues with the spacecraft.
After weeks of behind-the-scene recriminations, Roskosmos quietly launched an official probe into the contamination problem. According to a Directive of the Roskosmos Head No. VP-902-pr issued on July 11, 2013, a special commission had been set up to investigate problems in the MLM's propulsion system. GKNPTs Khrunichev, where MLM had been built, then issued its own Order No. 379 on July 23, 2013, to deal with the problem.
Throughout the complex labyrinth of pneumatic and hydraulic lines of the propulsion system, known as PGS DU, investigators found metallic particles ranging from 0.2 to 0.8 millimeters in size, which are orders of magnitude larger than any foreign objects allowable inside the system. The amount of contamination was also far beyond acceptable levels. The investigation officially confirmed that widespread contamination had caused multiple valves in the propulsion system to leak.
In fact, the only parts of the propulsion system on the MLM not affected by the contamination were the propellant transit lines for refueling of the Russian ISS segment from the Progress cargo ships and a sub-set of thrusters which MLM would use to help the ISS to conduct roll maneuvers.
MLM sent back to the assembly shop
As the extent of the problem plaguing MLM became apparent, it was decided to return the spacecraft to GKNPTs Khrunichev's assembly shop No. 22 for repairs. On Oct. 22, 2013, the Interfax news agency reported that it would take a year and a half to do all the fixes on the MLM at Khrunichev. According to industry sources at the time, the revised plans called for the launch of the MLM module in September 2015. The head of RKK Energia Vitaly Lopota told the RIA Novosti news agency that no decision for the return of the module back to GKNPTs Khrunichev had been made yet but, at the same time, Lopota admitted that he had not certified the spacecraft for launch.
Behind the scene, the probe into the MLM contamination was officially ended with a final conclusion, No. 174/3-2013. The document formally confirmed that propellant lines and systems of the propulsion system onboard the MLM module had not met the sterility requirements. The contamination of the internal lines of the PGS DU system caused valve EKOD2 to leak, the investigation said.
How had this happened?
According to the official conclusion of the investigation, the presence of foreign particles in the MLM's propulsion system was the result of the work conducted at GKNPTs Khrunichev's assembly plant, RKZ.
At the time, workers at Khrunichev were cutting pipelines and removing other components of the module's propulsion system, in order to reconfigure it from its original role as a backup to the Zarya FGB module into the MLM. For example, a set of six tanks, which would be used for refueling of the ISS during the FGB mission, were removed from the exterior of the spacecraft in order to make room for scientific instruments and for the attachment of the European Robotic Arm, ERA.
The official conclusion of the probe said that the contamination had stemmed from the "lack of methodological and technological support for the operations of cutting pipeline connections in the pneumatic and hydraulic system, PGS, which was needed to guarantee the meeting of requirements for ensuring the sterility of the internal cavities in the pipelines and system hardware." It is essentially bureaucratic speak for letting metallic dust formed during sawing off the lines pour into the interior of the remaining components.
According to one legend circulating at GKNPTs Khrunichev, the workers who were sawing off pipelines from the module thought they were dismantling the entire spacecraft for scrap. That story would sound completely unbelievable if not for other almost as incredible incidents of carelessness, poor quality control and incompetence within the industry in recent years, such as the installing navigation sensors on a Proton rocket in the upside down position or loading a Block DM-03 space tug on another Proton with too much propellant.
On Dec. 5, 2013, the Head of Roskosmos issued Decision No. ON-1545-r, ordering the formation of a working group tasked to restore the propulsion system on the MLM module. On Dec. 27, 2013, Roskosmos followed up with Decision No. 101/01-771, officially transferring the MLM module back to GKNPTs Khrunichev for repairs.
On Jan. 10, 2014, the head of RKK Energia Vitaly Lopota told the official ITAR-TASS news agency that the MLM module had been returned to GKNPTs Khrunichev on Dec. 31, 2013.
The repair team decided to almost completely dismantle the contaminated propulsion system of the MLM module and replace all its components, including the attitude control thrusters, with the new systems.
However, a number of critical pieces of contaminated hardware were simply no longer available for the module, which had originally been assembled in the 1990s, using systems developed in the 1970s and the 1980s! In total, six irreplaceable components would have to be cleaned, refurbished and installed back onto the spacecraft, if it was ever make it to the launch pad:
New launch schedule
As the contamination crisis around the MLM was deepening during 2013, its launch was pushed back from the end of that year to April and then to June 2014. By the end of 2013, NASA documents indicated that the MLM module would not fly before November 2015.
To make matters worse, the European Space Agency, ESA, responsible for the ERA mechanical arm onboard the MLM module had enough with all the delays and resulting cost overruns. ESA reportedly refused funding for the program beginning from 2014 onwards. As a result, the Russian government would likely have to pick up the tab for all further cost increases in the project...
The MLM Nauka module during testing at RKK Energia test facility, KIS, in June 2013. Click to enlarge. Credit: RKK Energia
The development prototype, KS, of the Nauka module at RKK Energia's test facility, KIS, in Korolev in June 2013. Click to enlarge. Credit: RKK Energia
Workers at RKK Energia's experimental plant work with the European robotic arm on the development prototype, KS, of the Nauka module. Click to enlarge. Credit: RKK Energia
Red boxes protect maneuvering thrusters on the FGB-2 module. Click to enlarge. Copyright © 2001 Anatoly Zak
The MLM Nauka module arrives at RKK Energia's Checkout and Testing Station, KIS, in Korolev on Dec. 14, 2012. Click to enlarge. Credit: RKK Energia
In 2013, multiple valves onboard the MLM module were found to be leaky as a result of severe contamination with metallic dust.
The size and amount of dust found inside MLM's hydraulic and propulsion system was far beyond allowable tolerance levels.