The standard method of obtaining observing time with Herschel was from the regular Announcements of Opportunity and their associated Calls for Proposals, the last of which was the OT2 Call that closed in summer 2011. However, for urgent observing requests that could not reasonably be covered by the Calls for Proposals, or that could not have been anticipated when a Call for Proposals was open, a quantity of observing time was reserved for Director's Discretionary Time (DDT). This could be obtained, when the circumstances justified it, as a DDT proposal (non urgent) or, for urgent observations, a Target of Opportunity (ToO) request.
There were three Announcements of Opportunity, each consisting of a Guaranteed Time and an Open Time Call for Proposals. Approximately one third of all Herschel time was reserved as Guaranteed Time, mainly for the instrument consortia who built the instruments, with a smaller amount distributed among the Herschel Science Team members.
It was recognised pre-launch that it was essential to dedicate a large fraction of observing time early in science operations to a small number of core science programmes proposed by large consortia, which would serve to populate the Herschel observations database with large numbers of targets that could be scheduled quickly, as soon as observing modes were released. These programmes would serve as training for a substantial number of people in the community in proposal preparation under the guidance of scientists familiar with the Herschel instrumentation and its capabilities. To be classified as a Key Programme, a proposal needed to be for a minimum of 100 hours of observing time.
The Announcement of Opportunity was made on 1st February 2007, with a deadline of 4th April 2007 for Guaranteed Time Key Programme (KPGT) proposals. However, due to a total power failure both of primary and emergency power supply at ESAC on the morning of closure that took down all computer services across the site, it was decided to extend the deadline for 24h to 5th April 2007. The call for Open Time Key Programme (KPOT) proposals opened on July 5th and closed on 25th October 2007.
A total of 11258h of observing time were awarded, although in the end only 10826h were actually used (this difference is due to changes in the time estimator making observing more efficient and modifications to programmes based on in-flight experience, leading to Guaranteed Time originally assigned to KPGT proposals being transferred to other, later programmes). The list of approved programmes, 21 Guaranteed Time and 21 Open Time, is shown in:
http://herschel.esac.esa.int/Key_Programmes.shtml.
The sky distribution of Key Programme AORs is shown in Figure 3.8, showing the remarkably homogeneous coverage of the sky by the 11650 Key Programme AORs approved by HOTAC and entered at the end of Phase 2 of proposal entry, easing the potential problems of inhomogeneous availability of targets through the year making scheduling inefficient.
Figure 3.8. The sky distribution of AOR centres for the 11650 accepted Key Programme AORs. Although there is a concentration both in the Galactic Plane and at the Galactic Poles, the overall distribution is remarkably homogeneous.
The time awarded by HOTAC was 57% of the nominal Herschel science time available for the mission of 19776h, consisting of 93% of the nominal Guaranteed Time and 40% of the nominal Open Time for the mission.
Given that the Key Programme Call closed more than a year and a half before launch, the AORs were based only on a theoretical knowledge of their likely behaviour in space, obtained from laboratory testing. In reality it was accepted that the time estimates were likely to be within 10% only of the final in-flight values, with the possibility that times could either increase or decrease in flight as the observing modes were tested and refined, thus there was the *potential* for as much as 1000h change between the original HOTAC awards and the actual time required to execute the observations.
Some important changes in the way that observations were executed were made very quickly. For example, it was found early in the mission that the slew time estimator was extremely conservative and that around 50s could be saved on all slews. As scan maps were made as line scans, with large numbers of slews, this led to a substantial saving in overheads for scan maps and a considerable saving in the time required to execute a map when implemented in November 2009. Similarly, it was found that the settling time for individual pointings in raster maps was overly conservative, with a minimum 8s settling time and larger values for bigger raster steps applied during Key Programme submission; this was reduced to a flat 5s (and, later, 3s for slews between line scans), even before launch, with substantial savings to many programmes. In contrast, it was found necessary to modify the way that PACS spectroscopy AORs were taken, leading to a significant increase in observing time for some proposals. Similarly, changes in the HIFI sequencer could either increase or decrease observing time according to the sub-mode and the band required. And the change of PACS photometry from point-source photometry mode to mini-scanmaps made a fundamental change to the execution of a large fraction of all the approved AORs.
The first in-flight Announcement of Opportunity (AO1) was made on February 25th 2010. This distributed most of the remaining Guaranteed Time (555h), plus 6577h of Open Time, of which 4988h were Priority 1 ("guaranteed" execution) and 1589h were Priority 2 time (fillers). The Guaranteed Time (GT1) Call opened on February 25th 2010, with it submission deadline on March 31st 2010. The Open Time Call (OT1) opened on May 20th 2010, with a submission deadline of 22nd July 2010.
A total of 33 GT1 and 241 OT1 proposals were accepted for execution, of which 176 of the accepted OT1 proposals were awarded Priority 1 time. The list of approved programmes is shown in http://herschel.esac.esa.int/AO-1_Programmes.shtml.
As a matter of policy it was decided that, as fillers, no constraints could be accepted on Priority 2 observations. This meant that no observations with timing constraints could be accepted as Priority 2 (a timing constraint was a scheduling window, an orientation constraint, a follow-on, or any other kind of scheduling restriction other than a concatenation). Solar system targets were defined to be time-constrained by nature. Proposals with time-constraints had either to be ranked high enough to obtain Priority 1 time, or be rejected.
The Second Announcement of Opportunity (AO2) was made on April 7th 2011. This distributed the very small amount of remaining Guaranteed Time (362h) plus 7590h of Open Time, of which 3420h were Priority 1 ("guaranteed" execution) and 4170h were Priority 2 time (fillers, with an expected execution rate of around 18% - in fact, the final degree of completion of Priority 2 time was 48%). The Guaranteed Time (GT2) Call opened on April 7th 2011, with it submission deadline on May 12th 2011. The Open Time Call (OT2) opened on June 9th 2011, with a submission deadline of 15th September 2011.
A total of 32 GT2 and 373 OT2 proposals were accepted for execution, of which 181 of the accepted OT2 proposals were awarded Priority 1 time. The list of approved programmes is shown in http://herschel.esac.esa.int/AO-2_Programmes.shtml.
The Proposal Handling System (PHS) received a rapidly increasing load during the last few days before closure. Figure 3.9 shows the evolution of load during the last 30 hours before closure. The load peaked at one proposal submission every 15 seconds shortly before closure. Most of the load was due to re-submissions of already submitted proposals as users followed advice to make an initial security copy submission and then fine-tune it, re-submitting a new version after each revision. However, during the last hour before closure, newly submitted proposals were also arriving at a rate of approximately one every two minutes.
Figure 3.9. Proposal Handling System load at closure of the OT2 Call. Shortly before closure the system was receiving and processing a proposal submission every 15 seconds.
Comparison of system load at closure of the OT1 and OT2 Calls showed that they were remarkably similar (see Figure 3.10) shows the evolution of load during the period that submissions were being received. The pattern observed during the last two weeks of the Call is almost identical. The slight discrepancies in the load between the two Calls prior to this can be explained by the fact that the OT1 Call closed before the August holiday period and the OT2 Call closed after it.
Save for cases where a target had visibility issues, all OT1 and OT2 Priority 1 observations had to be completed by December 31st 2012 to ensure guaranteed execution by EoH. In reality, efficient scheduling meant that OT1 and OT2 Priority 1 scheduling was essentially complete by early December 2012, allowing bulk scheduling of Priority 2 observations to start early.
As EoH approached, the Herschel Mission Scientists undertook the task to review the approved to-be-executed observing programmes with the objective of identifying potential "gaps" that could and should be filled. The aim was to identify any science that Herschel must do to avoid leaving gaps in its legacy. After a careful review, only a small number of such "Must-Do" observations were identified, showing that the Herschel legacy was, to a very large degree, complete, so the process was opened to the general community. The announcement was made on 25 May 2012, with a 14 June 2012 deadline.
Only completely new programmes that had never been proposed to HOTAC and that were deemed to be of very high scientific priority were accepted for execution as Must Do programmes. A total of 107.6h from seven proposals was approved after technical assessment and duplication checking; 43% of the approved time was awarded to a single, Planck follow-up programme. The list of approved programmes is shown in http://herschel.esac.esa.int/Must-Do_Programmes.shtml.
Figure 3.10. Comparison of the Proposal Handling System load at closure of the OT1 and OT2 Calls. The pattern is almost identical, demonstrating that the Herschel community are creatures of habit!
Figure 3.11. Comparison of the fraction of time requested for each of the Herschel instruments (counting SPIRE PACS Parallel Mode as a separate instrument) as a function of the three major Calls for Proposals during the mission. The overall level of demand for the instruments changes very little from Call to Call although it is noticeable that demand for Parallel Mode decreased sharply after the Key Programme Call as it became evident that, for many programmes, it was more efficient to use SPIRE only. In flight knowledge of the instrument performance showed that Parallel Mode had only very limited applications for cosmological programmes due to the far shallower depth reached by the PACS component of the data.
The first Must Do observations were scheduled on OD-1236. Scheduling of Must Do was essentially complete at EoH, with only a tiny number of observations with visibility beyond EoH that were not executed.
With Boil-Off well delayed with respect to predictions, there was testimonial announcement of a third Guaranteed Time Call at the end of April 2013, with no fixed deadline. The aim was to top-up the database with additional observations as they were submitted and accepted to improve scheduling efficiency, which would be adversely affected if helium continued through towards the end of Spring 2013. As boil-off happened within two days of the announcement of the GT3 Call, although three proposals were received before boil-off made the Call redundant, no GT3 observations were scheduled.
It was expected that instrument demand might change through the mission as the community grew to know the instrumentation better and passed from survey programmes to follow-up. In fact though, the demand for instruments proved to be remarkably stable through the mission. Figure 3.11) shows the evolution of the demand for each intrument -- treating SPIRE and Parallel Mode as separate entities which, in practice, they were not -- while Figure 3.12) shows the equivalent plot for the different sub-modes. The biggest single change is the shift from Parallel Mode to SPIRE photometry as the performance of Parallel Mode and its issues (see Section 3.2.2) became better understood.
Figure 3.12. Comparison of the fraction of time requested for each of the Herschel sub-instruments as a function of the three major Calls for Proposals during the mission. The relative demand for the different sub-instruments was remarkably stable between the closure of the initial Call in 2007 and the final Call in 2011.
The standard method of obtaining observing time with Herschel was from the regular Announcements of Opportunity and their associated Calls for Proposals, the last of which was the OT2 Call that closed in summer 2011. However, for urgent observing requests that could not reasonably be covered by the Calls for Proposals, or that could not have been anticipated when a Call for Proposals was open, a quantity of observing time was reserved for Director's Discretionary Time (DDT). This could be obtained, when the circumstances justified it, as a DDT proposal (non urgent) or, for urgent observations, a Target of Opportunity (ToO) request.
Although 5% of all Herschel nominal science time (1000h) was "reserved" for DDT proposals, only 1.5% of the final Herschel observing time was awarded in the form of Director's Discretionary Time.
The Herschel Space Observatory was fundamentally an "off-line" mission and was designed as such, so reacting rapidly was problematic. "Off-line" means that, instead of being in permanent contact with Herschel there was a Daily TeleCommunications Period (DTCP), normally for 3 hours each day, when the data stored on board had to be downlinked and new observations uplinked. The DTCP was within a few hours of local midnight at the ground station (prime was New Norcia, back-up was Cebreros), as Herschel was always close to the Opposition Point in the sky and thus at optimal altitude to be contacted for a few hours either side of local midnight from any ground station. The DTCP time varied according to whether Planck or Herschel was visible to the antenna first - every few months their orbits crossed over and the lead satellite in the DTCP changed - and also to demands on Ground Station resources from other missions that sometimes forced the communications window to be shifted further from the optimum time, based on a strict hierachy of Operational priorities for mission ground station access.
The off-line nature of Herschel meant that everything had to be ready for uplink before the DTCP started. This DTCP start time defined an unbreachable barrier and everything was calculated backwards from this moment.
At each DTCP, observations are uplinked for execution from 27 to 51h ahead. This means that even if a Ground Station pass were missed due to a communication problem, there were always enough observations in the on-board memory for Herschel to continue operating until the end of the DTCP period of the following day. All processing time for the planning file of observations to prepare it for uplink had be added to this minimum 51h. In practice, only two DTCPs were missed during the entire mission, one due to a sudden, very heavy snowfall at the Cebreros ground station that left the antenna unusable and the other due to a technical issue at New Norcia.
Ground station access to Herschel was thus the fundamental limitation to the Herschel reaction time to urgent scheduling requests. Assuming that the science was important enough to trigger a re-planning, the reaction to an urgent scheduling request was totally dependent on whether or not the re-planned observations could be processed and ready in time for the start of the target ground station pass. If the answer is not clearly "yes", it was unlikely that the request would be accepted.
There were a series of factors that decided how quickly Herschel could react to an urgent scheduling request. These are described briefly below. The faster the reaction required, the bigger the effort that was required to achieve it and the greater the knock-on effects and risk to normal spacecraft operations that it entailed. In general, any change to the observing schedule made less than 3 weeks before execution of the observations required special treatment and had to be justified carefully. The schedule was only be changed once submitted to MOC if there was a contingency (an instrument problem, or an operational issue that would lead to a significant loss of observing time, or for a ToO); it was not changed for "routine" tweaking of AORs.
The bottom line is that Herschel could, in normal circumstances, only guarantee to react in 7 days from an urgent request but could react in 5 days in favourable circumstances and in 4 days in a best case scenario. In theory and by taking considerable risks, faster reaction times would have been possible in a genuine emergency (e.g. a Galactic supernova), these though were never applied except in the case of operational contingencies involving instrument problems, where the need to save observing time was primordial.
Most of the steps described below applied too to normal observations too although, of course, on a much less compressed schedule.
The decision to attempt a fast turnaround time for an urgent scheduling request was not taken lightly. MOC at Darmstadt have to confirm that they could process the re-delivered observing schedule in time (even in an emergency this would not be attempted without a minimum lead time of 6 hours between delivery of a revised schedule by HSC and the start of the DTCP and, in normal circumstances, a lead time of 8 hours was required, even for a fast re-plan) and the HSC Mission Planning Group had to agree to a delivery schedule that allowed MOC enough time to carry out the full processing and check procedure for the re-planned schedule.
ToO and DDT proposals are a sub-set of Director's Discretionary Time. There were specific considerations for ToO proposals that are dealt with below. DDT proposals were submitted through HSpot following the normal guidelines for Open Time proposals; the DDT Call was always open for submissions. The most recent version of the PDFLatex proposal submission form available on the HSC Website should have been used for the submission (available here: http://herschel.esac.esa.int/Tools.shtml), although only the sections that are relevant to a DDT proposal needed to be filled in. The length of the science case needed to be commensurate with the amount of time requested (i.e. a 3 page science case was not necessary for a single 15 minute observation). An essential part of the science case was a clear explanation of why the award of Director's Discretionary Time was justified, particularly to provide a convincing case why the proposed observations should take priority in the telescope schedule over other proposals that had already been approved by HOTAC. It was recommended that a Helpdesk ticket be opened in the DDT/ToO Department confirming submission of a DDT proposal for consideration to accompany submission through HSpot, as all communication about the proposal was be made via Helpdesk, so that the history of the request and its processing was recorded and available to everyone involved in its scheduling.
As for a normal Open Time proposal, a DDT proposal could be retrieved after submission, revised and re-submitted as often as is necessary. If a proposal was revised, the proposer was advised to notify the Project Scientis and the HSC, via Helpdesk, when the final version had been submitted, so that processing of the proposal could begin.
The Project Scientist would not start to evaluate a DDT proposal until all technical reports on the contents of the proposal had been received (visibility, feasibility, AOR design, etc). When he was satisfied that he had all the information necessary for his evaluation, he would contact the HOTAC Chair and a joint decision would be made on acceptance, although the Project Scientist reserved the right to consult relevant experts, if necessary, to reach this decision. In general a decision on acceptance of DDT proposals would be taken within 3-4 weeks unless urgent scheduling considerations were involved.
For any Director's Discretionary Time proposal the main criterion for approval, if the observations were found to be feasible, was "helium into science". The proposal had to demonstrate that it would produce high enough quality science to be scheduled ahead of other, previously approved observing proposals.
There were two methods. Our preferred method was "pre-approval", but ToO alerts could be triggered without pre-approval if the science case is compelling, the circumstances justified it and there was a demonstrated need to react quickly to avoid missing a major scientific opportunity.
This was our preferred method. A normal proposal was submitted in the regular Calls for Proposals and evaluated. The proposal should have included clear trigger criteria and an expected reaction time that allowed its feasibility to be assessed. If HOTAC approved the request, the HSC was required to carry out the observations if the trigger criterion was met and scheduling constraints permitted it.
![[Note]](../../admonitions/note.gif) | Note |
|---|---|
| Sometimes a request was simply not be feasible for operational reasons, such as having the wrong instrument active, or because of operational constraints. |
This method had various advantages, not least of which is that the PI knew in advance that the observations were approved and would be made if possible and would be propietary. The observations were also already available in the HSC database, technically checked and only needed to be activated, saving valuable time on activation.
If the PI was satisfied that the trigger conditions had been met and that the target was visible, he or she sent a Helpdesk ticket in the ToO Department, supplying all the necessary information to allow the observations to be taken and a short justification of the trigger. The box stating that the ToO is pre-approved by HOTAC was ticked and the proposal name given to identify it. When the ticket was sent an SMS message went direct to the Project Scientist and to the HSC, alerting them of the triggering.
Provided that the Project Scientist was satisfied that the trigger conditions had been met, the observations would automatically be scheduled at the first possible opportunity.
Not all ToOs could be anticipated. Sometimes something would happen that was too good an opportunity to miss; it may have been a newly discovered comet that would become bright in 6 months time, or a sudden and unexpected outburst of a known object. In this case things could be more complex and may be slower.
The PI was requested to fill out the same form in the http://herschel.esac.esa.int/esupport/ ToO Department. The alert had to be justified and the required observations either provided or, at very least in the case of extremely urgent alerts, described in enough detail that an expert at HSC could prepare them. When the ticket was sent, an SMS message went direct to the Project Scientist and to the HSC, alerting them of the triggering. No action to schedule observations was be taken at HSC until the Project Scientist had approved the request, but a fast assessment would always be made of a new proposal to get an idea of what turnaround cycle in approval and scheduling would be required, including an assessment of when the observations could potentially be scheduled; where very urgent observations were requested, the approval cycle would be accelerated as much as possible.
The Project Scientist's first reaction to a ToO request was usually to request a technical assessment of the observations and target visibility from HSC. This may have required some backwards and forwards iteration with the PI to get the observations right, so that the time impact could be assessed and the feasibility ensured. If the request was for observations of a Solar System Object (SSO) it may have been necessary for software support at the HSC to add the SSO ephemeris to HSpot. The Project Scientist sought the approval of the HOTAC Chair to add the observations to the schedule. Approval (or not) was then communicated by the Project Scientist to the PI in a reply to the Helpdesk ticket.
A ToO request could also be submitted directly via HSpot in the same way as a normal proposal, which helped in its rapid assment as it ensured that all documentation was immediately available in standard format and the proposal was in the HSC database. HSpot (since HSpot 6.2.0) generated an both and e-mail and an SMS alert of a submitted proposal, so the On Call staff at HSC and Project Scientist were aware of submission, even out of normal working hours. We recommended strongly that users send also a Helpdesk ticket to the ToO/DDT department notifying the HSC of submission when using this method.
The scientific justification for a ToO request had to be completed using the standard PDFLatex form as for normal OT2 proposals, giving a level of detail commensurate with the amount of time being requested and only filling in those sections that were relevant to the request. This form and the instructions for completing it are available here: http://herschel.esac.esa.int/Tools.shtml.
Once the observations were approved by the Project Scientist, they were processed. The target day for execution was identified in consultation with MOC. Observations were normally submitted to MOC 18-20 days in advance of execution and the scheduled drafted 3 weeks or more in advance of execution. This means that any request for an observation less than 3 weeks ahead required re-processing at HSC and, usually, at MOC too.
Observations already in the database -- i.e. pre-approved ToO requests -- would be linked to the correct instrument control software and time estimator version and released to the Mission Planners, who then proceeded to de-commit the observing schedule for the day so that it could be modified. Once a de-commit of a schedule occurred it was removed completely from the system, meaning that even if the same schedule were subsequently re-instated, it had be completely re-processed at MOC, hence a schedule is only de-committed when it was certain that a re-plan is necessary and feasible and could be processed at MOC in time for uplink.
Once approved, observations not in the database had to be submitted as a new proposal first to get them into the Operational database. This became a much quicker and simpler process if they had already been submitted to HSpot by the proposer, as this made them available in the Astronomer database (the Astronomer database could be modified by PIs and their authorised co-Users, as well as by HSC staff, whereas the Operational database could only be modified by HSC staff - the Operational database was a "clean" database and only included observations previously approved for execution).
The new observation(s) were fitted into the schedule and a revised schedule generated. The revised schedule went through a list of checks and was then submitted to the instrument team(s) and the Project Scientist for final approval; either could request revision to the schedule as drafted if they identified any problems or issues with it. Only when all parties are satisfied would the schedule be transmitted to MOC. The change in schedule modified the sequence of pointing commands for the spacecraft and so had a knock-on effect for the following day too because it affected the initial state of the spacecraft reaction wheels for the next observing day, meaning that MOC had to generate a new solution for obtaining the desired pointing for both days, even though HSC had only re-generated the schedule for the first of them.
The time required at HSC to process the observations and generate the new schedule depended on the complexity of the changes to the schedule and the availability of the specialists to approve the draft schedule (out of normal working hours this was normally on a "best efforts" basis) and any modifications requested: on occasions it required a full working day and, in difficult cases, even longer to complete.
When the new schedule was approved, it was transmitted to MOC by FTP. Initial processing was automatic. Once this was complete a member of the Flight Dynamics Team would continue processing interactively. If all the checks were passed (this was not a trivial exercise, particularly with observations close to spacecraft limits), the schedule was then passed on to a member of the MOC Mission Planning Team who generated the file containing all the telecommands that would be transmitted to the spacecraft. This was usually the slowest and most complex part of the processing chain due to the serious potential consequences of any error. This processing at MOC required a full working day and had be completed in advance of the DTCP; if the DTCP was early in the day, or at the weekend, this meant that the task had to be completed on the previous working day.
Essentially, the peculiarities of the Herschel mission compared to on-line missions and the series of effectively uncompressable response times at different stages in the processing chain provided a fundamental limit to reaction times in any re-planning scenario.