"Spontaneous" Loss of Connection to All Equipment

Sequence Generator SGPro Support
1

Several times over the last month, I will have a sequence running over night that crashes out. It can run for hours without a problem, imaging targets, managing meridian flips, change filters, etc. But then, it will suddenly stop working.

When I log on in the morning, the sequence is still active, and a dialogue box will be showing, telling me there was an equipment related error (eg - SBIG Error while caching cooler status). But the sequence has frozen and made no progress from that point of the error (I assume).

When I look at the logs, it appears that SGP is suddenly losing connectivity to everything (focuser, rotator, camera) simultaneously. I don’t believe this is a cable issue, as each device is independently connected directly to an on OTA computer, the cables take separate routes, and, more tellingly, everything works fine if I abort the sequence and restart.

Any ideas on what might be at issue here and how I can avoid this? I happened to be watching when things went bad last night (at about 17 minutes after mindnight), so was able to just abort and restart the sequence from my office (no need to interact with the mount physically), and it went on to run without a hitch for the rest of the night, but on the previous two occasions, I lost a lot of imaging time.

Link to the log is:

https://1drv.ms/u/s!AsHXUvMw3K77gc0nmfp6n5tKzG4Grg?e=h27SDO

Useful Info

OS: Windows 10 PRO
Ver: <4.0.0.617

2

Could be power management on your computer turning everything “off”? On Windows you have to dive deep into the USB settings to full disable all power management activity. What OS are you running?

3

Windows 10 Pro.

I have done the deep dive on power management, and that is does not seem to be the issue (this is an Eagle3, and so power management is off by default, but I checked anyway).

4

And just to double check we’re talking about the same settings, this is where I had to disable USB power management to prevent my devices being disconnected:

usb_power
usb_power1288×784 119 KB

5

Same place for me. On the Eagle computer, there is no distinction between “On Battery” and “Plugged In”, there is just Enabled or Disabled. I have it Disabled, and always run from the mainlines, not a battery. Just for the sake of it, I just toggled back to enabled and disabled it again.

I do agree that everything turning off at once sounds like a computer going to sleep, but that doesn’t appear to be the case, as other utilities I have connected to the same device show no interruption…

6

You also need to check that in “Computer Management \ Device Manager \ Universal Serial Bus Controllers” for each device, on the Power Management tab, if present, that the two check boxes are un-ticked…

I’ve found that some updates will re-enable them…

7

Do I really I need to do that? One of the features of the Eagle3 is the ability to use software to turn the power to those ports on and off as a way to reset devices.

The settings in the power management tab of the USB controllers, to the best of my understanding, simply enable the ability to cut power to those ports. They don’t actually tell the computer to cut that power. Those settings are in the power management plans setting, aren’t they?

And isn’t the fact that other software programs are able to remain connected to these devices an indication that the problem is not with the USB port?

8

Which other programs are able to remain connected?

In your initial email you said that if you abort and restart everything works, but could that just be because the system has come out of power management mode…?

9

The mount logger runs continuously and shows no break in communications. And the software controller for the focuser works. That latter might be because I am waking the system up, but the settings in the system already should prevent any kind of shut down. I’m going to try the suggestion on preventing those USB connections from shutting down, but PrimaLuce has confirmed that this doesn’t happen on the Eagles (and I don’t do updates to the windows version they installed).

10

And the software controller for the focuser works

That’s interesting, as I’m sure as you saw in the logs, the initial failure was:

[12/15/20 00:17:02.787][DEBUG][CP Update Thread][SQ;] ASCOM Focuser: Error in GetCurrentPosition. : The port is closed.

Which external focuser s/w are you using? Wonder if it connects via ASCOM or native drivers? It’s possible that it’s just more robust to errors (e.g. maybe it just auto re-connects if connection is lost…?)

Anyway, will be interesting to see if disabling the USB device power-down mode fixes it. (I also have that disabled on my system)

11

What I suggested, is ‘lower level’ than the ‘higher level’ software control and drives the hardware directly…

Have a search on https://stargazerslounge.com/ where there are more posts on this ‘subject.’

12

I was imaging last night on 4.0.0.651 and had Focuser and EFW disconnect twice. Never had an issue before but have had lots of funnies happening since upgrading to this Beta.

13

I started with the current beta, so don;t have a basis for comparison. I am struggling with the unpredictability of it. I have confirmed my cables are all tight, and that the power is staying on. I don;t have hub issues, because I am using an Eagle3, but some nights the whole system blows up as SGP reports that nothing is connected, and some nights it runs smoothly. I have uploaded my logs, but no real insight yet on what could be happening.

14

I don’t know how your cables plug into the Eagle3, but any logic to following … odd that it looks like your filter wheel and autoguider remained connected. Just your Flat box, camera and NightCrawler were out.

I had a similar problem with my NightCrawler losing connection - as it turned out it was only near meridian limits, and was a bad cable. When it lost connection, it would knock out my autoguider for some reason. My little NUC has two separate USB banks (front and back), and the NC and AutoGuider were plugged into same bank. I swapped the cable to the NightCrawler, and fixed it - bizzare.

Also, out of curiosity, why did your telescope try to meridian flip in the middle of imaging the Horsehead - but then telescope reports its pointing EAST when you resume imaging the Horsehead after your ABORT?

Untitled-1
Untitled-12560×1080 811 KB

1st error
[12/15/20 00:17:02.787][DEBUG][CP Update Thread][SQ;] ASCOM Focuser: Error in GetCurrentPosition. : The port is closed.

… But 14 seconds later your telescope tried to flip …

[12/15/20 00:17:16.192][DEBUG][Sequence Thread][SQ;] Meridian Flip needed, Hour Angle >= Degrees Past To Flip: 4.4813827554056 >= 4.25
[12/15/20 00:17:16.192][DEBUG][Sequence Thread][SQ;] Running blocking meridian flip…
[12/15/20 00:17:16.192][DEBUG][Main Thread][SQ;] Adding sequence level notification: Running automatic pier flip…
[12/15/20 00:17:16.194][DEBUG][Sequence Thread][SQ;] Sending Notification: Status - Running automatic pier flip…

[12/15/20 00:17:16.301][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Starting Meridian Flip Procedure
[12/15/20 00:17:16.302][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Skipping Solve
[12/15/20 00:17:16.304][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Stopping the Auto Guider
[12/15/20 00:17:16.305][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Sending Telescope command to execute meridian flip
[12/15/20 00:17:16.496][DEBUG][Telescope Thread][SQ;MF;] ASCOM Telescope: Pier side is West
[12/15/20 00:17:16.496][DEBUG][Telescope Thread][SQ;MF;] ASCOM Telescope: attempting pier flip using slew

…
[12/15/20 00:17:17.356][INFO][Telescope Thread][SQ;MF;] Telescope: Using “OFFSET” sync option, updating slew cooridnates with offsets:
[12/15/20 00:17:17.356][INFO][Telescope Thread][SQ;MF;] RA: 0.378587786382307 hours
[12/15/20 00:17:17.356][INFO][Telescope Thread][SQ;MF;] DEC: -2.40640973578573 degrees
[12/15/20 00:17:17.356][DEBUG][Telescope Thread][SQ;MF;] Telescope: Slewing to J2000 RA: 5.67881679573461 (05h40m43.74s) Dec: -2.40640973578573 (-02°24’23.08")

…
[12/15/20 00:17:51.765][DEBUG][Telescope Thread][SQ;MF;] Telescope thread is IDLE…
[12/15/20 00:17:51.810][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Telescope command to meridian flip has completed
[12/15/20 00:17:51.811][DEBUG][Pier Flip Thread][SQ;MF;] Adjusting rotator for flip…
[12/15/20 00:17:51.811][DEBUG][Pier Flip Thread][SQ;MF;] Sequence is running, querying running target for target rotation info…
[12/15/20 00:17:51.811][DEBUG][Pier Flip Thread][SQ;MF;] Rotator is connected, but SGPro thinks it should not adjust camera angle…
[12/15/20 00:17:51.811][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Telescope has performed meridian flip
[12/15/20 00:17:51.811][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Flipping Auto Guider calibration data
[12/15/20 00:17:51.812][DEBUG][Pier Flip Thread][SQ;MF;] Autoguider (Direct Mount Guider) successfully flipped calibration data
[12/15/20 00:17:51.812][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Skipping auto center
[12/15/20 00:17:51.812][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Auto guider resuming
[12/15/20 00:17:51.812][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Auto guider resuming has completed

…

[12/15/20 00:17:58.735][DEBUG][Sequence Thread][SQ;] Meridian Flip successful
[12/15/20 00:17:58.739][DEBUG][Main Thread][SQ;] Adding sequence level notification: Automatic pier flip was successful…
[12/15/20 00:17:58.742][DEBUG][Pier Flip Thread][SQ;] Meridian Flip: Procedure complete
[12/15/20 00:17:58.742][DEBUG][Sequence Thread][SQ;] Sending Notification: StatusAutoFlip - Automatic pier flip was successful…
[12/15/20 00:17:58.742][DEBUG][Sequence Thread][SQ;] EventMarker(3) - t:Horsehead (2); e:1; f:19; o:
[12/15/20 00:17:58.742][DEBUG][Sequence Thread][SQ;] Waiting for AF to complete…
[12/15/20 00:17:58.751][DEBUG][Camera Thread][SQ;] SGM_FOCUSER_AUTO_FOCUS message received…
[12/15/20 00:17:58.751][DEBUG][Camera Thread][SQ;AF;] Alnitak Flat Box - Getting shutter status
[12/15/20 00:17:58.752][DEBUG][Camera Thread][SQ;AF;] Caught exception in AlnitakFlatBox:SendSerial: The port is closed.
[12/15/20 00:17:58.752][DEBUG][Camera Thread][SQ;AF;] Caught exception in AlnitakFlatBox:SendSerial: The port is closed.
[12/15/20 00:17:58.753][DEBUG][Camera Thread][SQ;AF;] Caught exception in AlnitakFlatBox:SendSerial: The port is closed.
[12/15/20 00:17:58.753][DEBUG][Camera Thread][SQ;AF;] Invalid return from serial. Val is ()

…

15

Thanks. I wonder about that, but all I need to do to get everything working is to exit SGP and restart the software. Doesn’t that indicate that this is not a cable issue?

As it happens, I replaced all the cables, but still had the issue. I will try plugging them in to different USB banks, but have been reluctant to do that, as only one of the banks has he ability to toggle power on and off.

Your software looks amazing. I am going to go download it immediately.

The meridian flip issue was an error created when I updated my mount ASCOM driver, to a version that included the side of pier command. I believe that my flip slew tolerance of 0 created an issue, but haven’t had a chance to confirm that (I reverted the driver back to the prior one which works fine).