Thanks, I do appreciate the sensible advice, and although I'll probably follow it in the short term for this particular system, there's more to it that I haven't explained yet, that means I'ld still like to find a solution that will work across different OS revisions if possible. Here's why:
I started putting together this cluster a few months ago with a few Pi 5's and half a dozen Intel PC's including all my old laptops, to help speed up an OCR project I was working on that was taking days to run, so with a mixture of Arm and Intel the easiest way to get them to work together was just to throw up some shell scripts to kick off jobs via ssh, with the compute nodes compiling their executables from source (cached of course) - and that worked well enough on a small scale for my purposes at the time. So I'm already comfortable with running a mixed cluster of different architectures and different levels of CPU power - my applications are all massively parallel so a slower system such as even the Pi Zero is not a bottleneck and can still usefully contribute some cycles.
I have three main applications that can use as much CPU as I can throw at them! One is a DNA assembler which works well at the moment on genomes the size of an amoeba but needs more nodes to scale up to human genome size; another is a new OCR program specifically for recovering computer source code from ancient listings, that gets better recognition than programs like tesseract but at the expense of some heavy CPU; and the third is a Scrabble algorithm I wrote some years ago with exhaustive searching at each level which wasn't able to run within competition time limits when I wrote it but might well manage it now...
One of those applications (the DNA assembler) was written for MPI so I thought it was time I put together a traditional cluster rather than continue with my hacky homebrew system.
I thought I might as well also add the other Pi's I have while I'm at it - I have a *lot* of Pi's of all models going back to the start of the Pi project - maybe 50 pi's ranging from the original model B though Pi 4's (though more Pi 3's than anything else), plus something like 60 Pi Zeros from another project - all of which I would like to add to this cluster eventually. (However a dozen or so of those systems are already in use doing various home automation tasks, so ideally I'ld like to avoid having to rebuild them from scratch. But they're underused in terms of CPU and could easily participate in a cluster.)
So although I was looking at adding Pi 4's next, soon I'll be adding those pre-aarch64 systems and the intel systems, so I'ld really like to find a way to make a heterogenous cluster work...
There may not be any shortcuts to doing this, but I thought if there were, the folks here would know them.
thanks,
Graham
I started putting together this cluster a few months ago with a few Pi 5's and half a dozen Intel PC's including all my old laptops, to help speed up an OCR project I was working on that was taking days to run, so with a mixture of Arm and Intel the easiest way to get them to work together was just to throw up some shell scripts to kick off jobs via ssh, with the compute nodes compiling their executables from source (cached of course) - and that worked well enough on a small scale for my purposes at the time. So I'm already comfortable with running a mixed cluster of different architectures and different levels of CPU power - my applications are all massively parallel so a slower system such as even the Pi Zero is not a bottleneck and can still usefully contribute some cycles.
I have three main applications that can use as much CPU as I can throw at them! One is a DNA assembler which works well at the moment on genomes the size of an amoeba but needs more nodes to scale up to human genome size; another is a new OCR program specifically for recovering computer source code from ancient listings, that gets better recognition than programs like tesseract but at the expense of some heavy CPU; and the third is a Scrabble algorithm I wrote some years ago with exhaustive searching at each level which wasn't able to run within competition time limits when I wrote it but might well manage it now...
One of those applications (the DNA assembler) was written for MPI so I thought it was time I put together a traditional cluster rather than continue with my hacky homebrew system.
I thought I might as well also add the other Pi's I have while I'm at it - I have a *lot* of Pi's of all models going back to the start of the Pi project - maybe 50 pi's ranging from the original model B though Pi 4's (though more Pi 3's than anything else), plus something like 60 Pi Zeros from another project - all of which I would like to add to this cluster eventually. (However a dozen or so of those systems are already in use doing various home automation tasks, so ideally I'ld like to avoid having to rebuild them from scratch. But they're underused in terms of CPU and could easily participate in a cluster.)
So although I was looking at adding Pi 4's next, soon I'll be adding those pre-aarch64 systems and the intel systems, so I'ld really like to find a way to make a heterogenous cluster work...
There may not be any shortcuts to doing this, but I thought if there were, the folks here would know them.
thanks,
Graham
Statistics: Posted by gtoal — Sun Jul 14, 2024 7:21 am