Problem with SGP handling failure of Alpaca Dome driver

Description

<I’m working with an astro society colleague to develop a custom ASCOM Alpaca Dome driver for his observatory dome. The intention is to use the driver as part of wider observation management controlled by Sequence Generator Pro.

The driver works fine with the ASCOM Device Hub and ConformU. Basic testing with Sequence Generator Pro is also looking very promising but testing potential Dome error conditions with SGP has shown up a couple of issues which are unexpected and unexplained so I’m seeking comments from SGP experts.

I’m using a basic two target sequence in SGP using the telescope, camera and filter wheel simulators as well as including my Alpaca Dome driver as an Observatory slaved to the telescope (in SGP).

This sequence works well when allowed to run to completion and interacts with my Dome driver and test rig as expected. To test some of the error handling, I set the Dome drivers ‘Az slew timeout’ setting such that the first ‘slewtoazimuth’ command sent to the dome driver would time out and raise an error.

By design, and in line with ASCOM General Principles, the dome driver supports ‘slewtoazimuth’ as an asynchronous process. Since a dome Az timeout could arise from a variety of hardware or software failures, my driver regards Az slew timeout as a fatal error leaving the dome driver in a potentially ‘compromised’ state. To communicate the condition to clients the ASCOM Alpaca protocol requires the driver to return Alpaca Internal Server Error (HTTP 500) in response to the next command from the client. ASCOM hands this back to the client as an exception. And, because the Az slew failure potentially leaves the driver in an ‘undefined’ state, my driver continues to return an Internal Server Error response for all subsequent commands from the client.

Observed Test Issue -

  • SGP starts its sequence and the Dome driver correctly responds to the expected tranche of calls to the Dome capability and status properties.
  • SGP commands the Dome shutter to open which operates correctly with valid responses to ASCOM/client.
  • SGP then commands the Az slew which the Dome driver correctly starts.

So all is well until the Az slew timeout happens.

  • Following the Az slew timeout the next command received by the dome from SGP was a call to get the ‘shutterstatus’ value.
  • The Dome driver responded with the required HTTP Response code of 500 (Internal Server Error) and a Response body type of text/html containing a textual description describing the Slew timeout error.
  • The ASCOM platform log provided shows the error at around 18:16:00 and it being raised as an exception back to the SGP client and the SGP log also shows an exception

I would expect this exception - which reflects a potentially serious failure on the Dome - to alert the user to the problem.

HOWEVER, SGP didn’t report the error in its GUI (even with the Notifications pane open and with Alert, Warning & Error checkboxes set in Sequence Generator Options). Instead it continued to send ‘Azimuth’ requests to the dome (each of which was processed by ASCOM and the Dome driver as above). This continued for several minutes with no sign of an error message appearing in SGP at which point I aborted the Sequence.

Following the Az slew timeout event, SGP continues to poll the ‘azimuth’ property at approx 1 second intervals and the SGP log continues to record the driver error returned from these calls. Weirdly, these are accompanied in the SGP log by entries of the form -

[11/12/22 18:16:43.644][DEBUG][Sequence Thread][SQ;] Telescope: Observatory is reporting slewing

However, there is no evidence of corresponding calls to the ‘slewing’ property in either the ASCOM platform logs or in my dome driver logs so its not clear here what SGP is doing.

So in a nutshell, it appeared as if SGP was waiting for the slew to complete and was polling the azimuth property optimistically. In doing so it appeared to be completely ignoring the exceptions raised by the Dome and passed back to SGP by the ASCOM platform.

Is there a way to have this error handled by SGP such that a human can intervene??

As mentioned, other than the above, SGP seems to work fine with my driver as long as no errors arise. Any pertinent observations would be appreciated.

SGP and ASCOM log files and sequence file are available on Dropbox and any other required files can be provided if required. I’ve also dropped a screen shot on Dropbox of what SGP was showing in the Notifications pane shortly after the slew timeout. >

Link to Logs

https://www.dropbox.com/sh/0rj0guk5niehkeu/AABbQTm3hMdKc5GnxB5nKpUya?dl=0

Useful Info

OS: <Microsoft Windows 8.1 Pro>
Ver: <SGP v4.1.0.765 (64-bit)>

Currently we are not handling exceptions coming out of the azimuth like that. I guess we could. We’d probably just abort the sequence at that point as there’s not much else we can do since it’s just a generic exception that’s coming back.

Jared

Hi Jared
Thank you for this clear and prompt reply.

I agree these sort of generic exceptions are difficult to manage at a more granular level with specific actions by SGP. But they can arise from events which can compromise an entire imaging run - eg in this case dome pointing in the wrong direction and cant be closed.

Surely that makes them more important to - at the very least - have them appear in the SGP Notifictions pane and - where used - appear in Email / SMS / Good Night System alerts etc.

Is there any prospect that this could be supported?

Regards
john