The Technology Experience Lab at NTT Global Data Centers provides data center managers and their teams the room and infrastructure to explore innovative approaches to building servers and storage for their clients. It enables businesses to evaluate the efficacy of private or hybrid clouds, distributed architectures, and alternative methods of delivering IT services in a low-risk environment. Its community of users also provides support and ideas through meetups, boot camps, webinars, and hackathons.
Back in 2017, there were several challenges that needed to be faced:- The desired net storage had to have 102TB.
- The raw storage had to have 240TB.
- The system had to use ZFS.
- The software had to offer high availability, no single point of failure, and high flexibility, with consistent snapshots and instant restore points when required.
- The hardware to support it had to be reliable and high-performing to take full advantage of the software and had to support several iSCSI block storage targets ranging from 10TB to 40TB, some with shared file folders.
.PDF file, 2.8 MB
The Chosen Solution
To find something that matched all the requirements, NTT Global Data Centers turned to Open-E’s partner, Toshiba, for a solution. In order to form the 102TB of net storage necessary, the team selected Toshiba’s 4 TB, 3.5″ SAS Enterprise Capacity Drives. Open-E JovianDSS was chosen as the supporting software. The MTTF (mean time to failure) of the hardware, 1,400,000 hours, and a non-recoverable error rate of just ten errors per 1016 bits read were ideal for achieving the reliability required. The 7,200 rpm drives achieved a Zpool read performance rating of 12.9 x single disk and 8.5 x single disk for writes. For the ZFS write logs and read caches, reliable 10 DWPD, SAS Enterprise SSDs with 1.6 TB storage from KIOXIA (formerly Toshiba Memory) were selected.
High Availability Failover Cluster Solution
To oppose the HDDs’ heat, AIC’s J4060-01 Dual Expander, 12 GB/s, 60 bay toploader JBOD was chosen. The 1400 W 1+1 hot-swap redundant 80+ Platinum power supply ensured electrical efficiency while also fulfilling the reliability requirements.The four 80 x 38 mm hot-swap fans in the JBOD made certain that the system never showed more than just a 4°C difference between the coolest and warmest drive temperatures. The Microchip Adaptec® RAID Adapter ASR-8885 provided connectivity to the JBOD with 8 internal and 8 external ports, run in HBA mode. This model was highlighted as a top choice back in 2017.
"The Technology Experience Lab benefits a lot from this storage solution, and we are truly impressed by its continuous stability. Set up within our highly available data centers, I can’t imagine a more reliable way of running enterprise storage. We’re looking forward to many more years of failure-free operation"
Dominik Friedel, Business Development Manager of NTT Global Data Centers
Achieved Results
The system has been running failure-free for five years now and is still going. The plan is to continue running the system to explore its reliability further – even a year after the supplier warranty ends for most of the enterprise-grade hardware components. Based on over 20 years of Open-E’s general experience as well as its specific experience gained in the field with Toshiba Enterprise HDDs, the team expects that the storage unit will have many more years of failure-free functionality to offer its owner.
"We are exceptionally proud to have the partner network of Open-E, KIOXIA, AIC, NTT and Microchip. Even if your Toshiba Hard Disk Drives are incredibly reliable, they still need a reliable system built out of many components to deliver real end-user value"
Rainer Kaese, Senior Manager Business Development, Storage Products Division, Toshiba Electronics Europe GmbH
"I’m proud to see how well this installation works after all these years, but I am also sure that this is not all this setup can do. Excellent performance and reliability are ensured by the ZFS and Linux-based system in combination with first-class hardware, so it can last for decades. And that’s not just a supposition; Open-E has been developing ZFS data storage systems for over 20 years and we have similar, long-lasting installations like this in use by customers all over the world"
Krzysztof Franek, CEO of Open-E, Inc.
Hardware details
| Configuration | |
|---|---|
| Server: | Supermicro X10 Series Server, 2U |
| Power supply: | 1400 W 1+1 hot-swap redundant 80+ Platinum power |
| Processor: | 2x Intel® Xeon® Series CPU |
| RAM: | 128GB RAM, Registered ECC |
| HBA: | 8 x internal 8 x external ports Microchip Adaptec® RAID Adapter ASR-8885 |
| Network interfaces: | 10Gbit/sec RJ45 Ethernet, 2 x Supermicro AOC-STG-I2T Network Card, 10Gbit/sec RJ45 Ethernet, X540 chip |
| Read Cache: | 2x 1.6TB Enterprise SSD |
| Write Cache (ZFS Intent Log): | 2x 1.6TB Enterprise SSD |
Storage configuration
| Licence: | Open-E JovianDSS |
| Features: | Standard HA Cluster Feature Pack |
| JBOD: | AIC’s J4060-01 Dual Expander, 12 GB/s, 60 bay toploader |
| HDD: | 60x Toshiba HDD MG04SCA40EA, 4TB 7200rpm SAS 12GB/s |
| zPool: | 30 x 2-way Mirror (2 Discs per group) |
| Spare storage capacity: | 10x KIOXIA DWPD SAS Enterprise SSDs with 1.6 TB |
