Single-Instance Executable, TSR-style programs, "lockfiles" and the DSA

Single-Instance Executable, TSR-style programs, "lockfiles" and the DSA

[Shark8]

I'm currently engaged in writing a series of programs for some scientists to control a few cameras; one system is a sort of cobbled together web-program, distributed across several computers [PHP for interface + C++ for mesage-slinging and camera-control], while the other is a single computer basically running the camera's manufacturer's program. -- This is mostly about the latter, though as the former will need to be addressed [via DSA(?)] in the near future.

There is another system that I'm not touching (for now) which uses lockfiles; sometimes (crashes and erroneous shutdowns) will leave the lockfiles behind. I have a controlled type-wrapper that will close its file if it is still open, and that could easily be adapted to delete them on finalization in the case of a lockfile. -- (#1) What is the best way that the community has come up with regarding lockfiles or similar functionality?

In this particular case, the lockfile represents that the control software for a particular instrument is already running, which brought to mind the old DOS TSRs where you would boot up the program and could call it (or another program using its services) again to achieve some different/special effects, which then brought to mind the new single-instance executables. Now, obviously the DSA can be used in this manner so that one partition provides services and the client partition queries/quits as needed. -- (#2)  Is there a non-DSA, and hopefully portable, Ada way to achieve single-instance executables? [I haven't had any luck trying web-searches on this topic.]

Re: Single-Instance Executable, TSR-style programs, "lockfiles" and the DSA

[Emmanuel Briot]

When I wrote the program that processes all incoming email on the mailing lists at AdaCore (in particular to manage tickets), we were using lockfiles indeed to coordinate between all the instances of the program (one per incoming email). The lockfile contained an expiration date and the PID of the process that took the lock, and a program was allowed to break the lock when that date was reached (like 10min or something, I forgot the value we came up with, when processing one message takes a few milliseconds), or when the process no longer existed (so crashed). So at least the system could not totally break and would eventually recover.

This is of course far from perfect, since during those 10 minutes no email could be processed or delivered, and if the timeout is incorrect we could end up with two programs executing concurrently (in practice, this was not a major issue for us and we could deal with the once-a-year duplicate ticket generated).

Years later, we finally moved to an actual database (postgresql) and we were able to remove the locks altogether by taking advantage of transactions there. This is of course a much better approach.

When I look at systems like Kafka (multi-node exchange of messages), they have an external program (ZooKeeper) in charge of monitoring the various instances. Presumably a similar approach could be used, where the external program is much simpler and only in charge of synchronizing things. Being simpler and fully written in Ada, it would be simpler to ensure this one doesn't crash (famous last words...).

Re: Single-Instance Executable, TSR-style programs, "lockfiles" and the DSA

[Shark8]

On Friday, September 10, 2021 at 1:10:51 AM UTC-6, briot.e wrote:
> When I wrote the program that processes all incoming email on the mailing lists at AdaCore (in particular to manage tickets), we were using lockfiles indeed to coordinate between all the instances of the program (one per incoming email). The lockfile contained an expiration date and the PID of the process that took the lock, and a program was allowed to break the lock when that date was reached (like 10min or something, I forgot the value we came up with, when processing one message takes a few milliseconds), or when the process no longer existed (so crashed). So at least the system could not totally break and would eventually recover. 
> 
> This is of course far from perfect, since during those 10 minutes no email could be processed or delivered, and if the timeout is incorrect we could end up with two programs executing concurrently (in practice, this was not a major issue for us and we could deal with the once-a-year duplicate ticket generated). 
> 
> Years later, we finally moved to an actual database (postgresql) and we were able to remove the locks altogether by taking advantage of transactions there. This is of course a much better approach.
Interesting.
When you moved to DB, did you use the DSA to have a Database-interface partition and client-query/-interface partition? I'm assuming not, because such a ticketing system probably doesn't have enough need for distributed clients, report-generators, etc. to justify such a design.

> When I look at systems like Kafka (multi-node exchange of messages), they have an external program (ZooKeeper) in charge of monitoring the various instances. Presumably a similar approach could be used, where the external program is much simpler and only in charge of synchronizing things. Being simpler and fully written in Ada, it would be simpler to ensure this one doesn't crash (famous last words...).
I suspect such designs are consequences of the poor support for processes/tasking that C has; the Ada equivalent of the functionality would be to have a TASK dedicated to the DB-interfacing [assuming single-node]; for full multi-node DB-backed/-transacted message-exchange DSA makes a lot of sense: Partition your DB-interface into a single node, then have your clients remote-interface that node.

Re: Single-Instance Executable, TSR-style programs, "lockfiles" and the DSA

[Emmanuel Briot]

> When you moved to DB, did you use the DSA to have a Database-interface partition and client-query/-interface partition? I'm assuming not, because such a ticketing system probably doesn't have enough need for distributed clients, report-generators, etc. to justify such a design.

We did not use the DSA. Postgres itself is a very capable server, which is implemented way more efficiently (and tested way better) than we could ever do I think, since this was only a side job. No reason to add an extra layer between the mail-processing program and the database.

> I suspect such designs are consequences of the poor support for processes/tasking that C has; the Ada equivalent of the functionality would be to have a TASK dedicated to the DB-interfacing [assuming single-node]; for full multi-node DB-backed/-transacted message-exchange DSA makes a lot of sense: Partition your DB-interface into a single node, then have your clients remote-interface that node.

I don't think you need a task dedicated to the database. Postgres handles concurrency very efficiently, it can do asynchronous queries if you really need that, and so on.
If you indeed have a database in your application, you could also use that to handle inter-process locking (pg_advisory_lock() for instance)