The College of Engineering operates a high performance computing (HPC) cluster that is supported by the Engineering community.  Our goal is to provide a resource that is available to all Engineering faculty and students.  The College supports the cluster by providing rack space, disk space, administration, and configuration of all systems, along with standard software.  Furthermore, the College provides CPU and GPU systems for general research and class use.  Faculty that purchase systems to add to the cluster get the benefit of reserved resources along with access to the rest of the cluster.  COE IT maintains these resources to free faculty up from system administration needs. 

 

About the HPC Cluster

The College of Engineering HPC cluster is a heterogeneous mix of 180 servers providing nearly 4000 CPU cores, over 140 GPUs, and over 36 TB total RAM.  The systems are connected via gigabit ethernet, and most of the latest servers also utilize a Mellanox EDR InfiniBand network connection.   The cluster also has access to 100TB global scratch from the College of Engineering's Dell/EMC Isilon enterprise storage.  The CoE HPC Cluster is rated at over 900 peak TFLOPS (double-precision).

The Hardware

Summary of HPC cluster:

  • 3 submit nodes
  • 6 Nvidia DGX2 nodes w/ 48 CPU cores, 16 V100 GPUs, and 1.5 TB RAM each
  • 3 Nvidia DGXS nodes w/ 20 CPU cores, 4 V100 GPUs, and 256 GB RAM each
  • 6 Dell AMD EPYC nodes w/ 48-64 CPU cores, 2 A40 GPUs, and 256 GB RAM each
  • 44 Dell Intel Skylake compute nodes, including:
    • 3 Dell DSS8440 node w/ 40+ CPU cores, 8 RTX8000 GPUs and 768 GB RAM
    • 1 Dell DSS8440 node w/ 40 CPU cores, 8 RTX6000 GPUs and 384 GB RAM
  • 25 Dell & Custom Intel Haswell/Broadwell compute nodes
  • 25 Dell Intel Sandy-bridge/Ivy-bridge compute nodes
  • 62 Dell Intel Westmere compute nodes
  • 4 Custom AMD Opteron compute nodes
  • more details below!

 

Submit Hosts (submit-a, b, c)

Model  Dell R740
Processor  Dual Processor 16 Core 2.10 Ghz Intel Xeon Gold 6130 with 22528 KB cache
Memory  256 GB RAM
Scratch storage  n/a
NIC

 4x Broadcom NetXtreme-E dual 10-Gigabit Ethernet Adapter

 Mellanox EDR InfiniBand

 

6 Nvidia DGX2 nodes (dgx2-[1-6])

Model  Nvidia DGX2
Processor  2x 24-core 2.70 Ghz Intel Xeon Platinum 8168 CPU with 33792 KB cache
GPUs  16 Nvidia Tesla V100 w/ 32GB
Memory  1.5 TB RAM @2666 MT/s
Scratch storage  28 TB NVME 
NIC

 Mellanox 100G ethernet

 Mellanox EDR Infiniband

 

3 Nvidia DGXS nodes (dgxs-[1-3])

Model  Nvidia DGXS
Processor  20-core 2.20 Ghz Intel Xeon E5-2698 with 51200 KB cache
GPUs  4 Nvidia Tesla V100 w/ 32GB
Memory  256 GB RAM @2400 MT/s
Scratch storage  1.5 TB NVME
NIC

 2x Intel X550T 10-Gigabit Ethernet Adapter

 

6 AMD EPYC compute nodes:

cn-r-[5-6]

Model  2x Dell PowerEdge R7525
Processor  2x 32-core 2.6 GHz AMD EPYC 7513 w/ 32768 KB cache
GPUs  2x Nvidia A40 w/ 48 GB
Memory  256 GB RAM @3200 MT/s
Scratch storage  725 GB
NIC

 Broadcom BCM5720 NetXtreme-E Ethernet Adapter

 Mellanox ConnectX-6 Infiniband

cn-r-[1-4]

Model  4x Dell PowerEdge R7525
Processor  2x 24-core 3.2 GHz AMD EPYC 7F72 w/ 16384 KB cache
GPUs  2x Nvidia A40 w/ 48 GB
Memory  256 GB RAM @3200 MT/s
Scratch storage  725 GB
NIC

 Broadcom BCM5720 NetXtreme-E Ethernet Adapter

 Mellanox ConnectX-6 Infiniband

 

44 Skylake compute nodes:

soundwave

Model  Dell PowerEdge R740xd2
Processor  2x 10-core 2.40 GHz Intel Xeon Silver 4210R w/ 14080 KB cache
Memory  384 GB RAM @2666 MT/s
Scratch storage  5.3 TB SSD
NIC

 Broadcom NetXtreme BCM5720 

 Mellanox ConnectX-5 Infiniband

optimus

Model  Dell PowerEdge DSS8440
Processor  2x 20-core 2.10 GHz Intel Xeon Gold 6230 w/ 28160 KB cache
GPUs  8x Nvidia Quadro RTX 6000 w/ 22GB 
Memory  384 GB RAM @2933 MT/s
Scratch storage  345 GB SSD
NIC

 Intel X710 10-Gigabit Ethernet Adapter

 Mellanox ConnectX-5 Infiniband

cn-gpu[6-7]

Model  Dell PowerEdge DSS8440
Processor  2x 24-core 3.0 GHz Intel Xeon Gold 6248R w/ 36608 KB cache
GPUs  8x Nvidia Quadro RTX 8000 w/ 44 GB 
Memory  768 GB RAM @2933 MT/s
Scratch storage  7.0 TB SSD
NIC

 Intel X710 10-Gigabit Ethernet Adapter

 Mellanox ConnectX-5 Infiniband

cn-gpu5

Model  Dell PowerEdge DSS8440
Processor  2x 20-core 2.50 GHz Intel Xeon Gold 6248 w/ 28160 KB cache
GPUs  8x Nvidia Quadro RTX 8000 w/ 44 GB 
Memory  768 GB RAM @2933 MT/s
Scratch storage  7.0 TB SSD
NIC

 Intel X710 10-Gigabit Ethernet Adapter

 Mellanox ConnectX-5 Infiniband

cn-q-[1-2]

Model  2x Dell PowerEdge R640
Processor  2x 24-core 3.0 GHz Intel Xeon Gold 6248R w/ 36608 KB cache
Memory  192 GB RAM @2933 MT/s
Scratch storage  345 GB
NIC

 Broadcom BCM5720 NetXtreme-E 10-Gigabit Ethernet

 Mellanox ConnectX-6 Infiniband

cn-p-1

Model  Dell PowerEdge R740xd
Processor  2x 24-core 3.0 GHz Intel Xeon Gold 6254R w/ 36608 KB cache
GPUs  1x Nvidia Tesla V100S w/ 32 GB
Memory  384 GB RAM @2933 MT/s
Scratch storage  345 GB
NIC

 Broadcom BCM5720 NetXtreme-E Ethernet Adapter

 Mellanox ConnectX-6 Infiniband

cn-o-1​​​​

Model  Dell PowerEdge R740
Processor  2x 24-core 3.0 GHz Intel Xeon Gold 6254R w/ 36608 KB cache
Memory  192 GB RAM @2933 MT/s
Scratch storage  345 GB
NIC

 Broadcom BCM5720 NetXtreme-E 10-Gigabit Ethernet

 Mellanox ConnectX-6 Infiniband

cn-n-[1-6]

Model  6x Dell PowerEdge R640
Processor  2x 18-core 3.10 GHz Intel Xeon Gold 6254 w/ 25344 KB cache
Memory  192 GB RAM @2933 MT/s
Scratch storage  345 GB
NIC

 Broadcom BCM57416 NetXtreme-E

 Mellanox ConnectX-5 Infiniband

cn-m-1

Model  Dell PowerEdge R740xd
Processor  2x 4-core 3.80 GHz Intel Xeon Gold 5222 w/ 16896 KB cache
GPUs  6x Nvidia Tesla T4 w/ 15 GB
Memory  192 GB RAM @2933 MT/s
Scratch storage  155 GB SSD
NIC

 Broadcom NetXtreme BCM5720 Ethernet Adapter

 Mellanox ConnectX-3 Infiniband

cn-m-2

Model  Dell Precision 7920
Processor  2x 8-core 2.50 GHz Intel Xeon Silver 4215 w/ 11264 KB cache
GPUs  2x Nvidia Quadro RTX 6000 w/ 24 GB
Memory  192 GB RAM @2666 MT/s
Scratch storage  165 GB SSD
NIC

 Intel I350 Gigabit Ethernet Adapter

 Mellanox ConnectX-5 Infiniband

cn-k-[1-4], cn-l-[1-2]

Model  6x Dell PowerEdge R740
Processor  2x 10-core 2.20 GHz Intel Xeon Silver 4114 w/ 14080 KB cache
Memory  64 GB RAM @2666 MT/s
Scratch storage  319-699 GB
NIC

 Broadcom BCM57416 NetXtreme-E 10-Gigabit Ethernet

 Mellanox ConnectX-5 Infiniband

cn-j-[1-4]

Model  4x Dell PowerEdge R740
Processor  2x 16-core 2.10 GHz Intel Xeon Gold 6130 w/ 22528 KB cache
Memory  512 GB RAM @2666 MT/s
Scratch storage  346 GB
NIC

 Broadcom BCM57416 NetXtreme-E 10-Gigabit Ethernet

 Mellanox ConnectX-3 Infiniband

cn-i-[4-5]

Model  2x Dell PowerEdge R740
Processor  2x 12-core 2.60 GHz Intel Xeon Gold 6126 w/ 19712 KB cache
Memory  256 GB RAM @2666 MT/s
Scratch storage  259 GB
NIC

 Broadcom BCM57416 NetXtreme-E 10-Gigabit Ethernet

 Mellanox ConnectX-3 Infiniband

cn-i-[1-3]

Model  3x Dell PowerEdge R740
Processor  2x 12-core 2.10 GHz Intel Xeon Silver 4116 w/ 16896 KB cache
Memory  128 GB RAM @2666 MT/s
Scratch storage  282 GB
NIC

 Intel X550T 10-Gigabit Ethernet

 Mellanox ConnectX-3 Infiniband

cn-h-[5-8]

Model  4x Dell PowerEdge R740
Processor  2x 22-core 2.10 GHz Intel Xeon Gold 6152 w/ 30976 KB cache
Memory  128 GB RAM @2666 MT/s
Scratch storage  346 GB
NIC

 Mellanox ConnectX-4 Ethernet

 Mellanox ConnectX Infiniband

cn-h-[1-4]

Model  4x Dell PowerEdge R740
Processor  2x 10-core 2.20 GHz Intel Xeon Silver 4114 w/ 16896 KB cache
Memory  128 GB RAM @2666 MT/s
Scratch storage  203 GB
NIC

 Broadcom BCM57416 NetXtreme-E 10-Gigabit Ethernet

 Mellanox ConnectX-3 Infiniband

cn-g-[1-4]

Model  4x Dell PowerEdge R640
Processor  2x 14-core 2.20 GHz Intel Xeon Gold 5120 w/ 19712 KB cache
Memory  64-128 GB RAM @2666 MT/s
Scratch storage  259 GB
NIC

 Broadcom BCM57416 NetXtreme-E 10-Gigabit Ethernet

 Mellanox ConnectX Infiniband

 

21 Dell Intel Haswell/Broadwell compute nodes:

cn-gpu0

Model  Dell PowerEdge R730
Processor  2x 12-core 2.30 GHz Intel Xeon E5-2670v3 w/ 30720 KB cache
GPUs  2x Nvidia Tesla M60 w/ 8GB
Memory  256 GB RAM @2133 MT/s
Scratch storage  3.8 TB
NIC

 Broadcom BCM57416 NetXtreme-E 10-Gigabit Ethernet

 Mellanox ConnectX-3 Infiniband

cn-f-[1-5]

Model  5x Dell PowerEdge R730
Processor  2x 10-core 2.60 GHz Intel Xeon E5-2660v3 w/ 25600 KB cache
Memory  128 GB RAM @2133 MT/s
Scratch storage  203 GB
NIC

 Broadcom BCM57800 NetXtreme II 1/10-Gigabit Ethernet

 Mellanox ConnectX-3 Infiniband

cn-e-[1-8]

Model  8x Dell PowerEdge R730
Processor  2x 14-core 2.60 GHz Intel Xeon E5-2660v3 w/ 25600 KB cache
Memory  32 GB RAM @2133 MT/s
Scratch storage  193 GB
NIC

 Broadcom BCM57800 NetXtreme II 1/10-Gigabit Ethernet

 Mellanox ConnectX-5 Infiniband

cn-d-[1-2]

Model  2x Dell PowerEdge R930
Processor  2x 14-core 2.00 GHz Intel Xeon E7-4830v4 w/ 35840 KB cache
Memory  384 GB RAM @2400 MT/s
Scratch storage  203 GB
NIC

 Broadcom BCM57800 NetXtreme II 1/10-Gigabit Ethernet

 Mellanox ConnectX-4 Infiniband

cn-d-[3-5]

Model  3x Dell PowerEdge R930
Processor  2x 20-core 2.10 GHz Intel Xeon E7-8870v4 w/ 51200 KB cache
Memory  512-1536 GB RAM @2400 MT/s
Scratch storage  203 GB
NIC

 Broadcom BCM57800 NetXtreme II 1/10-Gigabit Ethernet

 Mellanox ConnectX-4 Infiniband

cn-c-[11-12]

Model  2x Dell PowerEdge C6320
Processor  2x 8-core 2.40 GHz Intel Xeon E5-2630v3 w/ 20480 KB cache
Memory  128-256 GB RAM @1866 MT/s
Scratch storage  757 GB
NIC

 Intel 82599ES 10-Gigabit Ethernet

 Mellanox ConnectX-3 Infiniband

 

25 Dell Intel Sandy-bridge/Ivy-bridge compute nodes:

cn-b-[1-6], cn-c-[01-06]

Model  12x Dell PowerEdge C6320
Processor  2x 8-core 2.60 GHz Intel Xeon E5-2650v2 w/ 20480 KB cache
Memory  64-96 GB RAM @1866 MT/s
Scratch storage  338 GB
NIC

 Intel I350 Gigabit Ethernet

 Mellanox ConnectX Infiniband

cn-esa-[1-7]

Model  7x Dell PowerEdge R620
Processor  2x 8-core 2.90 GHz Intel Xeon E5-2690 w/ 20480 KB cache
Memory  64-128 GB RAM @1600 MT/s
Scratch storage  203 GB
NIC

 Broadcom NetXtreme BCM5720 Gigabit Ethernet

cn-a-[1-2]

Model  2x Dell PowerEdge R620
Processor  2x 8-core 2.60 GHz Intel Xeon E5-2670 w/ 20480 KB cache
Memory  128-256 GB RAM @1333 MT/s
Scratch storage  203 GB
NIC

 Broadcom NetXtreme BCM5720 Gigabit Ethernet

 Mellanox ConnectX Infiniband

cn-9-[1-4]

Model  4x Dell PowerEdge R620
Processor  2x 6-core 2.50 GHz Intel Xeon E5-2640 w/ 15360 KB cache
Memory  64-96 GB RAM @1333 MT/s
Scratch storage  147 GB
NIC

 Broadcom NetXtreme BCM5720 Gigabit Ethernet

 

62 Dell Nehalem/Westmere compute nodes (cn-0-[1-9], cn-1-[1-13], cn-2-[1-12], cn-3-[1-12], cn-4-[1-5], cn-5-[1-12])

Model  Dell PowerEdge R610
Processor  2x 6-core 2.67 GHz Intel Xeon X5650 w/ 12288KB cache
Memory  48-192 GB SDRAM
Hard Drive  160 GB SATA with Perc6
NIC

 4x Broadcom NetXtreme II Gigabit Ethernet Adapter

4 AMD Opteron 6172 compute nodes (cn-6-[1-4]):

Model   Custom 
Processor  Quad Processor 12 Core 2.1 Ghz AMD Opteron 6172 w/ 512 KB Cache
Memory  128 GB RAM
Hard Drive  160 GB SATA
NIC

 2x Intel 82576 Gigabit Ethernet Adapter

 

Software

The cluster is running Centos Linux 7.9 operating system.

The batch queue environment is Slurm Workload Manager.

Popular software include:

Matlab (2020b, 2021b)

Mathematica (11.0, 12.3.1)

Ansys (2019r3, 2020r2, 2021r2)

Intel Cluster Parallel Studio (2017-2020)

GCC 5.x - 11.x

Cuda 9.x, 10.x, 11.x