[NBLUG/talk] Multiple writes to the same file.

[Eric Eisenhart]

On Thu, Dec 23, 2004 at 11:38:32AM -0800, Walter Hansen wrote:
> Very good. Now I just need to work out a method of communication between
> the 50 processes. I did some socket work, but in my experiments sending a
> message to socket 1000 gets read by one of the proceses and not all. This
> works great to control my daemon, but the 50 need to be able to send a
> message that all of them can read.

Sounds to me like you need POE and you just don't know it yet.
http://poe.perl.org/

> I did just stumble accross IPC::Shareable which sets up special varriables
> that multiple processes can read to and write from. I'm thinking this
> might be the 'way' as then I can set up one array that each process could
> remove elements from on the fly and the other processes would be working
> with the same data. I do wonder what would happen if one process removes
> the last element and another process tries to access it (after all, it was
> there a .00002 sec ago). Race condition? Oh, well. I'll read more. I don't
> expect to have this up and running this year and the current 50 process
> works pretty well.

Sounds to me like what you're trying to implement is the classic
"Worker Pool" multi-threaded/multi-process setup.  You have "n" jobs to do
and want to run "m" workers at once to process them (where there's more jobs
than workers).  You also sound like either your workers or the controller
can discover new work to be done, which needs to be put onto the job queue
for the next available worker to handle.

This kind of programming is tricky to figure out the first time around. 
There's all sorts of subtle little issues you can run into, like deadlocks. 
Might be better to find working examples or modules that do most of the work
for you.

In order to avoid the race condition, you need to have some kind of
lock/semaphore.  IPC::Shareable has lock and unlock mechanisms.  You'd need
to lock the array before reading and unlock when you're totally done. 
(shared read locks don't make sense with perl variables, since a read can
cause a write)

Also, take into account that you can simplify things significantly by having
a "controller".  Instead of having every process talk to every other
process, have every process talk to the master controller and have the
master controller talk to the individual processes.  Then you could use
IPC::Open3 and set it up so that the master has a pair of file handles for
each child.  (but each child only has STDOUT to send messages to the master
and STDIN to receive messages)  You'd need the children to communicate their
status and the master to track what they're each doing.  "RUN 1234 param1"
"STARTED 1234", "ADD 5678 param2" "ADD 9012 param3" "FINISHED 1234" "RUN
3456 param4", etc...  You'd need to carefully avoid blocking I/O in the
master, perhaps by using select.  It's okay if the children block.

Or you could use POE::Component::JobQueue ...
http://search.cpan.org/~rcaputo/POE-Component-JobQueue-0.5402/JobQueue.pm

How you'd use POE depends on some details, though.  POE is actually
cooperative multitasking implemented in a single perl process, where I/O
that would normally block turns into nonblocking I/O that checks if any
other threads have something to do.  If your workers really truly need to be
separately forked processes, check out POE::Wheel::Run
(http://search.cpan.org/~rcaputo/POE-0.3003/lib/POE/Wheel/Run.pm).

BTW, POE rocks.  :)

> Oh, hey, Eric it's Walter from UNIX 50A/B back in 93? Hope your doing well!

Oh, you're *that* Walter.  Somehow I'd had it in my head that you were a
different Walter that I knew about a decade ago.  (no, really, I did!)  I'm
doing alright.
-- 
Eric Eisenhart
NBLUG Co-Founder & Scribe
The North Bay Linux Users Group
http://nblug.org/
eric at nblug.org, IRC: Freiheit at freenode, AIM: falschfreiheit, ICQ: 48217244
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://nblug.org/pipermail/talk/attachments/20041223/ef5abc03/attachment.pgp