ASCOM and multithreading

Sequence Generator
1

Hi Ken, Hi Jared,

I am playing with an ASCOM focuser driver I wrote, as the “official” one I am using right now is not very stable (it freezes every now and then, sometimes for some seconds, sometimes forever). When testing, I found out that my driver has the same problems and I could trace this to the fact that SGP is querying the driver from different threads, without syncing the queries.

To better see what is happening I logged the entry and the returning points on the interface (only for those properties or methods which needs access to the hardware over the serial port), like this:

…
Public ReadOnly Property MaxStep() As Integer Implements IFocuserV2.MaxStep
  Get
    TL.LogMessage("MaxStep Get", myMaxPosition.ToString())
    Return myMaxPosition	' Maximum extent of the focuser, a constant. 
  End Get
End Property

Public Sub Move(Position As Integer) Implements IFocuserV2.Move
  TL.LogMessage("Move", "Entered. Target Position: " + Position.ToString())
  MoveTo(Position)
  TL.LogMessage("Move", "Returning")
End Sub

Public ReadOnly Property Position() As Integer Implements IFocuserV2.Position
  Get
    TL.LogMessage("Position Get", "Entering")
    GetPosition()
    TL.LogMessage("Position Get", "Returning: " + myPosition.ToString())
    Return myPosition	' Return the focuser position
  End Get
End Property
…

And this is a fragment of the ASCOM conversation:

…
15:03:50.335 Connected Get             True
.
15:03:50.335 Link Get                  True
.
15:03:50.367 Connected Get             True
.
15:03:50.367 Absolute Get              True
.
15:03:50.368 Halt                      Entered
15:03:50.378 Halt                      Returning
.
15:03:50.379 Temperature Get           Entering
15:03:50.395 Temperature Get           Returning: 2,38
.
15:03:50.395 Absolute Get              True
.
15:03:50.395 MaxStep Get               50000
.
15:03:50.396 MaxIncrement Get          25000
.
**15:03:50.506 Temperature Get           Entering**
**15:03:50.512 Position Get              Entering**
**15:03:50.555 Temperature Get           Returning: 2,38**
**15:03:50.555 Position Get              Returning: 25800**
.
…
.
15:04:07.235 Position Get              Entering
15:04:07.254 Position Get              Returning: 25800
.
**15:04:07.556 IsMoving Get              Entered**
**15:04:07.560 Position Get              Entering**
**15:04:07.573 IsMoving Get              Returning: False**
**15:04:07.573 Position Get              Entering**
**15:04:07.589 Position Get              Returning: 25800**
**15:04:07.606 Position Get              Returning: 25800**
.
15:04:07.606 MaxStep Get               50000
.
15:04:07.612 IsMoving Get              Entered
15:04:07.636 IsMoving Get              Returning: False
.
15:04:07.638 Move                      Entered. Target Position: 26100
15:04:07.670 Move                      Returning
.
15:04:07.672 Position Get              Entering
15:04:07.700 Position Get              Returning: 25804
.
15:04:07.794 Position Get              Entering
15:04:07.813 Position Get              Returning: 25816
.
15:04:07.814 MaxStep Get               50000
…

As you can see, there are interleaved accesses (like at 15:03:50.506 or at 15:04:07.556) , where SGP does not wait for the current request to get answered. This creates problems at the serial interface to the focuser hardware, as the very simple protocol is by no means thread safe.

For this play-driver I could easily solve all problems by putting a sync-lock around the Com-Port access procedure. But, as far as I understood, there is no requirement in ASCOM that a driver should be thread-safe. And I think most of them are not.

Shouldn’t SGP take care of this? I have the feeling that this question should be extended also to the other, non-ASCOM drivers.

Kind regards,
Horia

2

Hori,

Good point. I did a tooling for the mount and used as well ascot interfaces out of a multithreading system. Same experience. Final result was to concentrate all accesse to an ascom object through one wrapper, which in case does then thread locking. That definitely works.

Michel

3

SGP cannot be expected to handle making an ASCOM driver thread safe. Any driver has the ability to be accessed by multiple clients if the driver is constructed in such a way. Thus the driver needs to enforce it’s own thread safety (should it need it). For instance my Dome driver can handle overlapping requests because it’s not bound to a serial port but my focuser cannot as it is bound to a serial port. SGP does not know (and should not) know about these things or else we get into hardware specific implementation…and at that point what good is ASCOM?

Making a driver thread safe is almost trivial (as you mentioned). You just need a lock around the communication to keep things atomic. Here is a very simple example I wrote a while back. Look for instances of mutex

Most of them actually are…but if you want to make your driver not thread safe and still play nicely with SGP then you need to make the Move synchronous which will block SGP from querying other things. If move is ASYNC then we assume you can handle overlapping I/O as you’re not blocking…and effectively you MUST support overlapping I/O because the client has no way of knowing when the move is done without asking.

As for ASCOM’s take on it:
http://www.ascom-standards.org/Developer/Throttling.htm

A client application or script must be able to use the driver’s properties and methods without regard to timing.
If this were not the case, applications would need to be filled with device-specific code – delays and caching of values that depend on which telescope type it is using.

Thus, a driver must be able to take whatever an application can throw at it. There is nothing in any of the specifications that rate-limits an application and the driver must be able to handle any stream of property requests and method calls from the client, with any timing. If the hardware has timing issues, the driver must handle them (after all it knows the hardware and the client does not). Typically, then, a driver would need to have traffic throttling. This is also an extension of The General Principle.

Sorry if I seem “salty” we’ve just fought this fight many times :slight_smile:

Thanks,
Jared

4

Jared,

fully agreed. According to the ASCOM doc’s its fine. Not all are functioning - not your job.

Michel

5

Hi Jared,

Thank you for answering.

No problem with that, I actually have not had the impression you are. I live in Germany and being assertive is the standard.

I would not read that as a requirement for a driver to be thread safe. If the writer of the driver knows about multithreading and race hazards, he might have done something against, just because he knows. If not, well …

I can follow your decision to say SGP assumes all drivers are thread safe and I think this applies not only to ASCOM but also to the native drivers. I also understand that it is impossible for you to test SGP against all the available combinations. The problem I have, as a convinced user, is that sometimes, out of nothing, something stops working. And this usually happens during one of those perfect but rare nights.

Should then ASCOM issue a statement about thread safety, making it an explicit requirement? Should SGP issue a statement that it expects all drivers to be thread safe?

Kind regards,
Horia

6

I don’t think there is a “requirement” more of a recommendation or best practice. You can always run Conform against your driver too and see what that says. If you have the ability to easily protect your device why wouldn’t you want to do that?

This seems pretty explicit to me:

Client applications that use ASCOM drivers are becoming increasingly sophisticated in their use of multiple threads to provide a comprehensive and responsive user experience. This being the case, drivers authors have to ensure that their drivers will perform well if used by such applications.

For native drivers we have considerably more control so we get to dictate how we access those drivers. Any thread safety that those devices require is then “on us” to implement. We don’t have this control with ASCOM devices. Having said that I can’t think of any native device that we have locks around to be thread safe. I don’t believe any native device that we support requires this as they’re pretty much all cameras which don’t use a serial connection.

Thanks,
Jared

7

Yes, you are absolutely right. I have missed that.

Kind regards,
Horia

8

ASCOM drivers should be thread safe. The hardware probably isn’t, usually the problem is that a second serial command is sent before the reply to the first one is received.

The simplest fix is to call the same function for all serial commands and put a lock round the transmit and receive functions so if a second call to the hardware is made it blocks until the first one has completed.

Here’s an example:

   // object used for locking to prevent multiple drivers accessing common code at the same time
   private static readonly object lockObject = new object();

    /// <summary>
    /// Sends the message with at least replycounts bytes, reads additional characters to a # terminator
    /// </summary>
    /// <param name="message">The message.</param>
    /// <param name="replycounts">The minimum number of reply bytes.</param>
    /// <returns></returns>
    private static byte[] SendMessageCounted(byte[] message, int replycounts)
    {
        lock (lockObject)
        {
            try
            {
                Log.Message("SendMessageCounted", "send {0}, expect {1}", ToPrintableString(message), replycounts);
                serial.ClearBuffers();
                serial.TransmitBinary(message);
                var reply = new List<byte>(serial.ReceiveCountedBinary(replycounts));
                Log.Message("SendMessageCounted", " received {0}", ToPrintableString(reply.ToArray()));
                return reply.ToArray();
            }
            catch (Exception ex)
            {
                Log.Issue("SendMessageCounted", " error: {0}", ex.Message);
                throw;
            }
        }
    }

I’ve tended to move to a queue based system but this should be OK. BTW it’s a bit more complex than you need, there were other things with the reply that I needed to manage.

Chris

9

Thank you Chris, this is exactely how I solved it. I guess the most important was to understand that thread safety is required.

Kind regards,
Horia