Meridian flip failed and sequence aborted on Mach1 followed by mount pier collision

[Leo]

Hello,

I’m using A-P Mach1 mount with SGP for already few years and automatic meridian flip was working flawlessly except some minor glitch. Yesterday there was a disaster - the meridian flip was detected as needed at 1:13AM and the SGP rejected to perform the flip and aborted the sequence as shown in attached log section below. I went to sleep exactly 20 minutes before the flip and the rest of night was lost. Usually I’m taking a drive to my remote imaging location for just one time per month, so such loss of imaging hours is very disturbing to me. Besides failed sequence, the mount continued tracking until couple of hours later the telescope bumped into the tripod and caused motor stall condition. I hope no mechanical damage was done to the system. I’d like to understand why is it happening (I have the latest SGP Pro version 3.1.0.519 and latest version of PHD2) and what should I do to prevent that in the future and still have some sleep.

I’m also using A-P APCC Pro version 1.8.3.1. The APCC sends limit with offset to SGP Pro (checkbox is enabled), enabled meridian tracking limits, action when limit is reachedL just warned.

In SGP Sequence I have “Use Auto Meridian Flip” enabled with the following settings:

Minutes Past Meridian to Flip:1
Auto Center After Meridian Flip - Enabled
(all other options are disabled)
Auto meridian flip auto-close delay 10

BTW, the value of 1 minute as minutes past meridian to flip was probably set inadvertently. In my equipment profile setting I keep it at 0.

[07/24/20 01:13:35.650][DEBUG][Sequence Thread][SQ;] Meridian Flip needed, Hour Angle >= Degrees Past To Flip: -3.30657500000001 >= -3.5
[07/24/20 01:13:35.650][DEBUG][Sequence Thread][SQ;] Running blocking meridian flip…
[07/24/20 01:13:35.650][DEBUG][Main Thread][SQ;] Adding sequence level notification: Running automatic pier flip…
[07/24/20 01:13:35.662][DEBUG][Sequence Thread][SQ;] Sending Notification: Status - Running automatic pier flip…
[07/24/20 01:13:35.662][DEBUG][Sequence Thread][SQ;] Handling monitoring event (Good Night System, Status): (july23-2020) Running automatic pier flip…
[07/24/20 01:13:35.662][DEBUG][Sequence Thread][SQ;] GNS: Sent status message to GNS ((july23-2020) Running automatic pier flip…)…
[07/24/20 01:13:35.670][DEBUG][Main Thread][SQ;] Adding sequence level notification: Cannot Run Meridian Flip:
-Telescope must be pointed across the meridian to pier flip.

[07/24/20 01:13:35.717][DEBUG][Main Thread][SQ;] Sending Notification: Error - Cannot Run Meridian Flip:
-Telescope must be pointed across the meridian to pier flip.

[07/24/20 01:13:35.718][DEBUG][Main Thread][SQ;] Handling monitoring event (Good Night System, Error): (july23-2020) Cannot Run Meridian Flip:
-Telescope must be pointed across the meridian to pier flip.

[07/24/20 01:13:35.718][DEBUG][Main Thread][SQ;] GNS: Sent error message to GNS ((july23-2020) Cannot Run Meridian Flip:
-Telescope must be pointed across the meridian to pier flip.
)…
[07/24/20 01:13:35.721][DEBUG][Main Thread][SQ;] Adding sequence level notification: Failed to complete automatic meridian flip, aborting sequence!
[07/24/20 01:13:35.768][DEBUG][Sequence Thread][SQ;] Sending Notification: Error - Failed to complete automatic meridian flip, aborting sequence!
[07/24/20 01:13:35.768][DEBUG][Sequence Thread][SQ;] Handling monitoring event (Good Night System, Error): (july23-2020) Failed to complete automatic meridian flip, aborting sequence!

[Jared]

Can you attach the entire SGP log, I need more info than what is in that snippet. My initial guess is that the time either on the scope or on your machine is set incorrectly.

Thank you,
Jared

[Leo]

Jared, I’ve sent you full log in a private message. I initialized the mount using APCC’s Initialize Mount dialogue, which is using current computer time, and I was using GPS coordinates for current location (done all that many times before). I haven’t done anything unusual, which could have affected time settings. In fact, I run atomic clock application to sync my notebook time in precise way and it also matched my local time - just a day before that failed imaging session.

Mount doesn’t have a battery for keeping time, it’s getting time from PC and its controlling application, APCC, which, I suppose, is getting its time from PC clock.

I’ve also checked with SkyX that meridian crossing of my target was around 01:10AM and frame duration was 4 minutes.

Leo

[Leo]

Hello Jared,

I did some testing today with APCC Pro and SGP. I found that meridian offset is causing the same kind of problem that I’ve experienced in the field. Please see attached screenshot of the moment when SGP fails to perform the meridian flip. It looks like an error in SGP which assumes that the mount cannot perform meridian flip few moments prior to actual meridian cross (while APCC was setup specifically to allow flip prior to meridian). I guess that something similar happened in my last imaging session, although the offset was set to 0 (probably due to some residual error which caused the scope to point to east just a few seconds prior to meridian crossing).

I’ve also checked my APCC settings that it’s configured to initialize mount time from PC time (I’ve also checked mount time to be equal to PC time during the test), so the problem is not there.

My understanding that meridian offset in APCC was designed as a safety measure to stop mount tracking (or park it) certain time after controlling software (such as SGP) fails to perform meridian flip, but that doesn’t work at all and fails during testing. As alternative, I could use the SGP’s feature to stop tracking at sequence end, but it’s not my preferable option, since I start and stop sequence manually several times during imaging session and I don’t want mount tracking to stop in the meantime. Instead I’d like to have SGP to stop tracking by default as a safety measure when sequence is aborted for any reason.

In addition to said above, if SGP refused to initiate the meridian flip due to internal logic limiting it to time after meridian cross, the problem could have been avoided by continuing tracking just another few moments and trying to perform the flip after the meridian cross - before deciding to give up and abort the sequence.

In the attached screenshot you can see the SkyX’s yellow circle marking actual scope position relative to meridian shown as a red line, as well as all relevant meridian settings of APCC, and also the error messages by SGP, which are later followed by complete sequence abort.

[bryan]

Leo

Perhaps I misunderstand your description.

Meridian delay is intended to allow CW up tracking so that you can track from horizon to horizon without the need for a flip.

West Meridian Limits are safety limits, i.e. they prevent the scope from tracking into the pier on the west side with CW up.

From APCC documentation

he Meridian Limits are one of the most powerful and useful features of Astro-Physics mounts when used in conjunction with the Astro-Physics Command Center (APCC). They open up the possibility of imaging on both sides of the meridian WITHOUT wasting time on a pier flip at declinations where the scope will not strike the pier. The limits, when properly configured, allow you to track safely past the meridian up to the limits you set based on your unique instrument package. In addition, they allow you to begin an image ahead of the meridian by safely slewing you into a counterweight-up position to start an imaging series that can then simply continue through the meridian. However, like any powerful feature, you MUST understand how the meridian limits work. It is vital that you understand what they CANNOT do as well as understanding what they can do

Bryan

[EricC]

APCC can help prevent pier crashes but you have to set it up correctly. Change the “Just Warn” to “stop tracking” (or something like that). Then you need to set tracking limits in the graph area for different DEC values. See the documentation.
Eric

[Leo]

Bryan,

Meridian delay can be useful for several things, including possibility to track without flip, in certain cases where there won’t be pier crash. I don’t argue with that. I’ve had a session with Howard from A-P who helped me to setup meridian limits and also helped me to clarify certain things. I can now setup my system to avoid this kind of disaster, set some safety limits in APCC, but I still think that:

• SGP can be more robust to allow flexibility in waiting for proper time to allow pier flip - instead of aborting sequence. That would be useful for users who are not aware of possible timing accuracy issues with crossing meridian, especially when using default 0 delay.

• SGP could be improved by considering failed flip as kind of error, that will be followed by recovery attempt, based on its global error recovery policy (it’s also easy to verify that simple time pause will solve the problem and allow pier flip to go thru) using “Attempt to automatically recover the sequence - every xxx minutes for yyy min”

• In case that sequence finally fails after all attempts - stop mount tracking even if telescope options don’t specify park or “stop tracking”

• Mach1 mount allows pier flips before meridian crossing as long as they are performed within meridian limits defined by APCC. SGP currently doesn’t support pier flips prior to meridian regardless of these limits and claims that the mount doesn’t support flips prior to meridian crossing. This is probably a limitation by SGP which can be modified to take into account the meridian limits of APCC.

Leo

[Chris]

@Leo

• Mach1 mount allows pier flips before meridian crossing as long as they are performed within meridian limits defined by APCC. SGP currently doesn’t support pier flips prior to meridian regardless of these limits and claims that the mount doesn’t support flips prior to meridian crossing. This is probably a limitation by SGP which can be modified to take into account the meridian limits of APCC.

AIUI SGP will flip before the meridian by using the ASCOM set pier side command. The ASCOM driver must report CanSetPierSide as true and implement the PierSide set method. If set PierSide command is sent when the mount can’t do it then the mount should throw an exception.

I guess that the AP driver doesn’t do this but can be set so that the hour angle after which a slew will change the pier side is before the meridian.

One thing to remember is that SGP is written to work for a multitude of people using a wide variety of mounts. Expecting that it will handle edge cases for one particular mount and user is a big ask.

Expecting that the mount will protect itself and stop tracking before it runs into something seems reasonable but something that should really be implemented by the mount. That way the mount will stop if SGP crashes. Celestron mounts do this.

[bryan]

Thanks for the explanation, Leo.

Bryan

[Leo]

Hi Chris, thank you for reply. I’ve created a post on ap-gto forum on the issue of pier flip prior to meridian crossing.

[Fran]

Hello,

just one word to report that three nights ago I had the same issue with my EQ8 mount: when trying to run the flip pier after 3 hours of tracking and adquiring images, SGPro launched the famous “Something terrible has happened…”. SGPro runs fine normally, but this is the second time that this issue happens (no configuration changes from earlier nights where SGPro performed fine).

The flip pier is set to run when the scope reach the zenith meridian, no tracking after crossing meridian is programmed.

I’ll look for the log night if necessary.

Greetings,
Fran

[Leo]

For some reason, SGP does not perform automatic flip anymore, even when scope has passed meridian. Please see attached fragment of SGP log. I tried to restart PC, performed all standard steps including mount init, star alignment, etc. Nothing helps - SGP fails to perform either automatic or manual flip. On the other hand, I can run flip from APCC when pressing “Pier Flip” button manually.

[08/07/20 18:36:34.542][DEBUG][Sequence Thread][SQ;] Telescope is on the West side of the mount
[08/07/20 18:36:34.543][DEBUG][Sequence Thread][SQ;] Meridian Flip needed, Hour Angle >= Degrees Past To Flip: 3.97464999999999 >= 3.75
[08/07/20 18:36:34.543][DEBUG][Sequence Thread][SQ;] Running blocking meridian flip…
[08/07/20 18:36:34.543][DEBUG][Main Thread][SQ;] Adding sequence level notification: Running automatic pier flip…
[08/07/20 18:36:34.547][DEBUG][Sequence Thread][SQ;] Sending Notification: Status - Running automatic pier flip…
[08/07/20 18:36:34.547][DEBUG][Sequence Thread][SQ;] Handling monitoring event (Good Night System, Status): (fliptest) Running automatic pier flip…
[08/07/20 18:36:34.547][DEBUG][Sequence Thread][SQ;] GNS: Sent status message to GNS ((fliptest) Running automatic pier flip…)…
[08/07/20 18:36:34.653][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Starting Meridian Flip Procedure
[08/07/20 18:36:34.653][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Skipping Solve
[08/07/20 18:36:34.653][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Stopping the Auto Guider
[08/07/20 18:36:34.656][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Sending Telescope command to execute meridian flip
[08/07/20 18:36:34.670][DEBUG][Telescope Thread][SQ;MF;] ASCOM Telescope: Pier side is West
[08/07/20 18:36:34.670][DEBUG][Telescope Thread][SQ;MF;] ASCOM Telescope: attempting pier flip using slew
[08/07/20 18:36:34.720][DEBUG][Telescope Thread][SQ;MF;] Telescope: Slewing to J2000 RA: 14.7536210927312 (14h45m13.04s) Dec: 64.2599136085692 (64°15’35.69")
[08/07/20 18:36:34.720][DEBUG][Telescope Thread][SQ;MF;] Telescope: Slew received J2000 coordinates, mount requires JNOW, converting…
[08/07/20 18:36:34.721][DEBUG][Telescope Thread][SQ;MF;] Telescope: Slewing to JNOW RA: 14.7606388892779 Dec: 64.1805555555567
[08/07/20 18:36:41.121][DEBUG][Telescope Thread][SQ;MF;] Scope reports it is done with synchronous slew, verifying…
[08/07/20 18:36:41.153][DEBUG][Telescope Thread][SQ;MF;] Telescope: Slewing has completed
[08/07/20 18:36:41.153][DEBUG][Telescope Thread][SQ;MF;] Telescope: Settling for 5 seconds
[08/07/20 18:36:46.154][DEBUG][Telescope Thread][SQ;MF;] Telescope: Settling has completed
[08/07/20 18:36:46.667][DEBUG][Telescope Thread][SQ;MF;] ASCOM Telescope: Failed to flip because starting pier side and ending pier side are the same!
[08/07/20 18:36:46.941][DEBUG][Main Thread][SQ;MF;] PopulateDataModel: Transferring view to the data model…
[08/07/20 18:36:46.952][DEBUG][MF Update Thread][SQ;MF;] Performing serialize…
[08/07/20 18:36:47.168][DEBUG][Telescope Thread][SQ;MF;] Telescope thread is IDLE…
[08/07/20 18:36:47.208][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Telescope command to meridian flip has completed
[08/07/20 18:36:47.208][DEBUG][Pier Flip Thread][SQ;MF;] Meridian Flip: Telescope failed to perform meridian flip
[08/07/20 18:36:47.841][DEBUG][Main Thread][SQ;MF;] Meridian flip was user aborted…
[08/07/20 18:36:47.880][DEBUG][Main Thread][SQ;] Adding sequence level notification: Failed to complete automatic meridian flip, aborting sequence!
[08/07/20 18:36:47.920][DEBUG][Sequence Thread][SQ;] Sending Notification: Error - Failed to complete automatic meridian flip, aborting sequence!
[08/07/20 18:36:47.920][DEBUG][Sequence Thread][SQ;] Handling monitoring event (Good Night System, Error): (fliptest) Failed to complete automatic meridian flip, aborting sequence!
[08/07/20 18:36:47.920][DEBUG][Sequence Thread][SQ;] GNS: Sent error message to GNS ((fliptest) Failed to complete automatic meridian flip, aborting sequence!)…
[08/07/20 18:36:47.924][DEBUG][Pier Flip Thread][SQ;] Meridian Flip: Procedure complete
[08/07/20 18:36:50.915][DEBUG][Sequence Thread][SQ;] Blocking Pier Flip: Failed to meridian flip, aborting sequence (True)
[08/07/20 18:36:50.915][DEBUG][Sequence Thread][SQ;] Aborting sequence: Failed to meridian flip
[08/07/20 18:36:50.915][DEBUG][Sequence Thread][SQ;] Set sequence abort
[08/07/20 18:36:50.917][DEBUG][Main Thread][SQ;] Adding sequence level notification: Failed to meridian flip, aborting sequence.

[Ross_Walker]

@Fran

The flip pier is set to run when the scope reach the zenith meridian

But Fran, how many minutes after passing the meridian did you set SGP to do the flip? If you set it to zero then that is a huge mistake IHMO, and even one minute might be a bit edgy. In general, I would never set the “minutes after meridian” setting to zero because there can be slight differences between when SGP thinks the meridian has been reached and when your mount believes the meridian has been reached. If you set it to something like 3 to 5 minutes, there should be no ambiguity.

If you have the minutes after setting set to a suitably large number then ignore the above

[rodolgo]

Hi,
I experienced the same issue a few days ago: MACH1, APCC, no specific meridian limits in APCC, just the plain vanilla settings.
I was tracking M27, quite high in the sky at this time of the year. The ASCOM scope driver’s HA reading was approaching +12 hrs - the moubnt should have flipped just afterwards. The CWUP indicator was “lit” in the ASCOM driver interface. But it did not flip - had to perform a manual slew to “force” the flip.
Lucky I was present at the scope to avoid a crash.
Interestingly, the time to flip indicator in SGP status bar was empty (no reading) - as if it hadn’t recognized the need to flip.
The only recent change I can think of is the update to ASCOM Platform 6.
I’ll try to capture logs at the next imaging opportunity.

[rgralak]

Unless you have it configured to do so (not recommended), APCC will not flip the scope at the meridian. A slew has to be issued by a client application (e.g. SGPro) for the mount to potentially perform a flip.

-Ray

[rodolgo]

Thanks for the response, Ray.
This is also my expectation - SGP should have triggered the flip, as it usually does, but didn’t happen that time, for some obscure reason