How to connect multiple z840 workstations

[okurum]

Hello,

I have a Z840 with 2 x Xeon E5-2680 v4 @ 2.40GHz (28 Cores) and 224GB RAM used to run compiled Fortran codes for numerical simulations (mostly using OpenMPI). 

I want to add more (2 or 3) Z840s (found some refurbished ones) to scale the computational power up by connecting all machines together using InfiniBand. The motherboard supports PCIe v3.0 so I assume my options are limited and I do not have much knowledge on how to best achieve what I want. 

From what I could gather a QDR (40Gb) InfiniBand connection should be adequate. If someone has experience on how to best setup 3-4 Z840s connected together via InfiniBand, it would be very much appreciated. e.g., what's the best network card(s) to buy?, do I need a switch? or can machines be connected to each other directly? 

Thanks

[DGroves]

does your software support running multiple instances of itself on separate systems?

if not then a 3rd party application must be used to tie your application together

simply networking computers together will not make the software you run on them able to mesh/sync their data the software you are running must support this function

last, yes you will need a switch

https://www.open-mpi.org/papers/ipdps-2006/

https://www.researchgate.net/publication/4245345_Infiniband_scalability_in_Open_MPI

[okurum]

Hi DGroves

thanks for the reply. 
The open-source code I am using is compiled with gfortran+openmpi and can run on multiple nodes on a proper HPC cluster (e.g, 100s of cpu cores distributed over many nodes).  
Locally though the code (software) is developed and tested on one Z840 workstation. I am looking into increasing the “local” computational power by adding homogenous workstations to reduce the need to go to the HPC cluster. 

Very simply put, I want to increase 28 cores to 112 cores by adding 3 more Z840s. So it’s not about running multiple instances of the software on multiple nodes. I want to be able to run one instance of the software using the combined cpu/ram of 4 workstations. 

From what I’ve read so far a qdr infiniband card on each machine with a switch to connect them all will do this but I cannot be sure if I am missing something. 

I am looking for cost effective hardware advice (infiniband cards, switch, cables) that are compatible with the Z840s (supports PCIe v3.0). 

any further advice is appreciated. Cheers!

[DGroves]

what you are asking/wanting to do is  not a turnkey solution you will have to spend time to figure out how to configure the software to work in a multisystem configuration and at the same time hope your selection of hardware will be flexible/configurable for the required task.

i recommend you start doing quite a bit reading on this subject by reading any white papers the software vender has written on this subject and then also post the the venders online forum where other users of this software can reply