The SDSC Data Oasis is an extremely large scale parallel file system (PFS), with almost four petabytes of total disk capacity shared among various HPC resources at SDSC. The system provides between 60 and 120 gigabytes per second of data movement bandwidth and manages between 3000 and 6000 individual disks. TSCC has exclusive 10 GbE access to a minimum of 200 terabytes of raw disk space for use as a temporary, high bandwidth, very high capacity storage system for use while running jobs on the hotel, condo and PDAF clusters.
The purpose of TSCC's PFS is to provide running jobs with fast, parallel access to data, not long-term data storage. Files created on or copied to Data Oasis will belong to group scratch and will be deleted automatically after 90 days. Groups who are doing longer-running analyses on data sets should request an allocation and purge cycle that will meet their needs. Those needing a permanent place to store data should visit SDSC Storage & Backup Solutions or email SDSC Services to arrange project storage that can be accessed from TSCC.
To see the performance characteristics, backup policies, hardware, and data management software on Data Oasis and TSCC's other storage facilities, see the Data Storage Overview. View the Backup page for complete details.
Data Oasis is fully connected to all of the TSCC nodes, providing exceptional data movement and data management throughput to users of all three node configurations. The GPU nodes will also have full Data Oasis connectivity when configured.
Data Oasis continues to be enhanced and upgraded to improve capacity, reliability and performance. Subscribe to the TSCC Discussion List for up-to-the-minute notification about Data Oasis and all aspects of the Triton Shared Computing Cluster.
Data Oasis on TSCC is a Lustre-based parallel distributed high performance filesystem. It consists of 16 Object Storage Servers (OSS) and 64 Object Storage Targets (OST). Each OST has approximately 12.7 TB of usable storage, giving TSCC Data Oasis a total storage capacity of around 812.8 TB.
Data Oasis has its default stripe count set to 1 OST, with a default stripe size of one MB. The following command lists stripes of files or directories: lfs getstripe <file/dir-name>
You may consider striping larger files, e.g. over one MB, to multiple OSTs for better performance.
Striping is file-based, using a round-robin load-balancing algorithm. By default, a 100 MB file will arrive in 100 blocks at a single OST. The next file from the same client or process will transfer to the next OST, and so forth. This behavior can be changed using the lfs setstripe command.
The command /usr/sbin/lctl dl -t lists the layout of Oasis OSTs and their associated OSSs.
Although striping a file to many OSTs can increase performance, it may also decrease reliability.
Lustre is not efficient in dealing with many small files (a few KB or smaller in size). Listing or querying large numbers of files, such as with the ls -R and du commands can also be slow. It is recommended to use Lustre-optimized commands such as /usr/bin/lfs find instead whenever possible.
It is not recommended that users reference Data Oasis in their login processes. Lustre can be unstable from time to time.