77 TB of research data lost due to HPE software update
Kyoto University lost 77TB of critical research data from its supercomputer because Hewlett Packard Enterprise (HPE) released a software update that caused a script to malfunction and deleted the backup data. As a result, days of work are lost and a significant portion of erased data is lost forever.
Kyoto University lost around 34 million files from 14 research groups generated from December 14 to 16, according to The battery. GizChina reported that the university was unable to restore data from four groups by backup and is therefore gone forever. Initially, Kyoto specialists believed the university had lost up to 100TB, but the disaster limit turned out to be 77TB of data.
HPE pushed an update that caused a script to malfunction that deletes log files older than ten days. However, instead of deleting the old log files stored with the backups in a high capacity storage system, it erased all files from the backup instead, erasing 77TB of critical research data.
HPE admitted that its software update caused the problem and took 100% responsibility.
“From 5:32 p.m. on December 14, 2021 to 12:43 p.m. on December 16, 2021, due to a defect in the program that backs up the storage of the supercomputer system (made by Japan Hewlett Packard), the supercomputer system [malfunctioned]”A statement from HPE translated by Google reads.” As a result, an accident occurred in which some data in high capacity storage (/ LARGE0) was inadvertently deleted. […] The backup log from the past which was originally useless due to a problem of careless modification of the program and its application procedure in the repair function of the backup program by Japan Hewlett Packard, the supplier of the super system computer science. The file deletion process malfunctioned as did the file deletion process under the / LARGE0 directory. “
The team suspended the backup process on the supercomputer. Still, Kyoto University plans to resume backing up by the end of January after fixing the software and script glitch and taking steps to prevent a recurrence.