5.2. From proposal to observations and exploitation of the data archive

5.2.1. ICC and HSC user support

5.2.1.1. Routine mission support

From approximately three years before launch, when the first call for proposals was made, to the end of helium, the Herschel Science Centre, in collaboration with the ICCs, has provided the information required for the submission of proposals in the Herschel Space Observatory Web site (http://herschel.esac.esa.int). Its role has now switched in post-Operations to one of supporting efficient exploitation of the data archive.

Astronomers are requested to register to access observatory services, which included the capability to submit proposals and continue to provide access to the Helpdesk for the resolution of all issues related to data to Operations and for the retrieval and exploitation of observational data from the Herschel Science Archive. Only registered users can submit Helpdesk tickets, can be co-users of executed proposals and can retrieve data from the archive.

5.2.1.2. Winding-up of post-Operations activities

Support to users from the Herschel Science Centre will continue through to the end of the Archive Consolidation Phase, at the end of 2017, at which point it will become mainly the ongoing work of the ESA archives group, the Satellite Archives Team (SAT). The levels of support from the individual ICCs will drop to a low levels by 2015 and individual ICCs will cease to operate during 2016, with PACS being the first to wind up on March 31st 2016 and HIFI the last, on December 31st 2016. Formal support for the United States community will end with the winding-up of NHSC on September 30th 2017.

5.2.1.3. Herschel Helpdesk

The standard user interface to the Herschel Science Centre has been and will remain the Helpdesk (http://herschel.esac.esa.int/esupport/). As of the end of November 2013, 6626 questions have been dealt with through the HSC Helpdesk. However, as the average Helpdesk ticket leads to approximately five interactions (notifications, enquiry, reply to user, response from user, reply to response, etc) there have been more than 32 000 interactions with users in total through the Helpdesk. The system went live with the first question to the Herschel Helpdesk submitted on Monday 05 Feb 2007 at 06:54 PM (CET).

Helpdesk is a web-based system. This has numerous advantages. As tickets can only be generated by registered users through a web interface, the problem of wasted time and resources in filtering the genuine enquiries from users from spam messages is eliminated. It also makes it easy to identify the Herschel community and to avoid the need for a scattergun approach to user communications. The ticketing system also allows tickets to be assigned a priority on a six-level scale from "low", through "medium" (the standard level for the majority of communications) to "Critical" -- the highest level of urgency, used to flag issues that needed a reply with extreme urgency, for example, scheduling issues that threatened the integrity of observations, or the delivery of schedules to MOC. In post-operations it seems unlikely that any level higher than "Urgent" will be appropriate: this would be for reporting, for example, serious software issues for which an urgent workaround is needed.

Users need to be aware though that, although they receive notification of answers to Helpdesk tickets via e-mail, the return address (hschelp@sciops.esa.int) is a fictious, no-return address: any reply to this address will be undelivered. The user should click on the link in the e-mail and answer the ticket within the Helpdesk system.

A further advantage of the web-based system is that it helps tracking which enquiries are awaiting reply and how long they have been inactive in the system. HSC personnel have taken great pride in minimising the delay in responding to queries as much as possible, but a small fraction of tickets can be "lost in the cracks" -- it was possible to track easily through the Helpdesk system which tickets had not be answered on a day to day basis and take the necessary action, where required. The Helpdesk system would flag any urgent ticket for which a response was overdue.

Throughout operations, Helpdesk tickets were almost invariably read and assessed almost immediately almost 24h a day, even on weekends. Urgent tickets would generally receive a human response of some kind within an hour of submission, if only to reassure the user that a human being was at the end of the line and that the ticket had been seen and was being acted on. While genuinely urgent enquiries are less frequent in post-operations, assessment of new tickets is still carried out rapidly and, where necessary (although this process is usually not seen by the user) classified and brought to the attention of the appropriate expert. While complicated technical issues and software problem reports may take some time to answer adequately, particularly for instrument specialists who have, at times in Operations, become heavily overloaded, many routine enquires have been answered almost immediately throughout Operations, even outside of normal working hours; this service has been offered on a best efforts basis, based on the goodwill of HSC personnel and, only at critical junctures such as the closure of Calls, has it been formalised and then only for a maximum of 48 hours before closure. Even so, even a considerable fraction of routine enquiries have been dealt with in real time, or near real time, for around eighteen hours per day, seven days a week through operations.

Using Helpdesk for user interactions leaves a paper trail that allows responses to be tracked. This enables consistent responses to be given to common queries. However, tickets are not searchable from the Internet (i.e. you cannot Google a question and see a Herschel Helpdesk answer), so they remain private and confidential between the user and the HSC and can only be seen by HSC personnel with Helpdesk access rights.

During post-Operations the intensity of Helpdesk traffic has, logically, reduced considerably however, the typical level of traffic is still 30-40 interactions with users per week.

As post-Operations continue, the HSC workforce will be reduced progressively, with team members being increasingly shared with other missions. However, a full Helpdesk service will continue until the end of post-Operations, although there will be a logical reduction in the level of specialist instrument support, particularly from the start of 2015.

Herschel's Helpdesk will continue to function even after the end of post-Operations, but this will be on a best efforts basis, without specific staff assigned to answer queries.

5.2.2. Proposal preparation and submission

5.2.2.1. Proposal submission and retrieval

Proposal preparation and submission was done through the HSpot tool (see Section 6.2, “Introduction to HSpot”), the Herschel Observation Planning Software. A valid Herschel scientific proposal contained at least one AOR, or Astronomical Observation Request. Each AOR is based on an AOT, or Astronomical Observation Template, which is a pre-defined observing mode, characterised by an instrument configuration and way of operation that have been optimised for the execution of a particular type of observation (see Chapter 6, Observing with Herschel). An AOR was generated when the proposer provided the parameters required for the selected AOT to personalise it to his or her individual requirements, and is equivalent to the term "observation" used in this document.

A proposal submitted through HSpot is stored in the Herschel Space Observatory database. The proposer, and co-proposers selected by the principal investigator to be co-users, were allowed to retrieve, modify and upload their proposal(s) until the closing date of the Announcement of Opportunity. At that time, the database was closed to HSpot, and the HSC distributed the stored proposals to the HOTAC panels. Proposers could check the status of their proposal(s) in relation to the HOTAC review in the Proposal status Web page (http://herschel.esac.esa.int). During the review process, the HSC provided support to the HOTAC and, on request, assessed the technical feasibility of the observations. In addition, a systematic technical feasibility assessment is carried out on all accepted proposals to identify conflicts such as duplications of individual observations or complete proposals with existing or newly submitted proposals (if necessary, HOTAC would consider two or more similar proposals together), technical problems with proposals (observations that were badly, or not optimally designed, or even proposals that were not feasible technically).

All accepted proposals and their associated AORs are available through HSpot and may be downloaded and examined (but not modified and re-submitted) by any registered user. This was done initially to allow new users to design their own observations based on best practices used by the Guaranteed Time users who were generally instrument experts. During the post-Operations phase it may be useful to go back and examine how an observations retrieved from the Archive was designed and executed. The observation can be retrieved in HSpot and a user can set the specific date and time of execution to see exactly how the observation was performed by using the "AOR overlay" option and then animating the AOR.

5.2.2.2. Proposal review and updating

The period of proposal submission before the HOTAC review was called Phase-1. After the HOTAC review the results were made public once collated, checked and ratified by the ESA Director of Science. At this point proposal submission Phase-2 started. In this period, observers were expected to refine their accepted proposals, modifying them following the HOTAC guidelines (this included making any changes that were required after the technical revision of the AORs), remove duplications and prepare the set of AORs that would be executed with the observatory.

Proposals would not be released for execution until the necessary revisions had been made and validated successfully. This meant that the updated AOTs and the latest available observatory knowledge would be used for the observations that would be carried out. Please see the "Herschel Space Observatory Call for Proposals: Policies and Procedures" document for a definition of proposal submission Phase-1 and Phase-2, and for the policies on proposal modifications. The end of proposal submission Phase-2 resulted in a consolidated database of accepted proposals and its corresponding AORs.

5.2.2.3. Late changes to proposals

It was sometimes necessary for a PI to modify the AORs even after Phase 2 has ended. Possible reasons include correcting or optimising coordinates, changes of observing strategy as a result of analysing previous observations, forced changes due to instrumental issues that may have emerged (from time to time the HSC contacted PIs to warn them of instrumental issues that had been detected and to recommend changes of strategy) and the need to replace targets when the original target is found to be unsuitable in the light of new information.

Minor changes could be made by the PI at any time, although to guarantee that the revised AORs could be scheduled they were required at least 4 weeks and preferably 6 weeks before they are due to be executed to allow time for processing (all new AORs go through technical checks) and to allow for the lead time needed when advance drafts of the schedule are prepared. However, more substantial modifications (e.g. changes of target, or changes of observing mode) had to be approved by the Project Scientist before the AORs could be released for scheduling and, of necessity, had to be properly justified, with the exact modifications detailed; these changes had to be coherent with the original aims of the proposal as approved by HOTAC.

5.2.2.4. The post-Operations database

Logically, once EoH was reached, no more science AORs could be executed. It thus made no sense to retrieve and update proposals. Some engineering and a few calibration AORs were executed during the six weeks of post-cryo Operations before passivisation, although these are of no use to general users. Once passivisation had been completed and some clean-up of the Archive carried out (the elimination of observations from the database that were scheduled by the HSC and delivered to MOC, but never executed because of EoH) the database enters its final state and becomes frozen. Users can no longer modify proposals and AORs, but HSC personnel can still make some modifications in the database: this may still be necessary if, for example, subtle problems are identified in a particular observation or observations, necesitating that they be declared failed after detailed Quality Control follow-up.

5.2.3. Data retrieval

After execution, each Herschel observation has had its proprietary period, initially one year for Key Programmes, for which data reduction was more challenging as the quantities of data were greater and the knowledge of the instruments and their calibration correspondingly less. This proprietary period was later reduced to six months. Some observations approved as DDT/ToO have had a shorter proprietary period, or even none at all, if it was deemed that the observations were being performed as a public service and should be made available to any interested user. After the proprietary period has ended, the observations become public. The last data to become public were those from the end of the mission, whose proprietary period ended at the end of October 2013, six months after the end of helium. From the start of November 2013 all Herschel science observations have been public and can be consulted and accessed by all registered users.

While proprietary, only the PI of the data and designated co-users can see browse products or retrieve the data. For proprietary data, retrieval is blocked and the browse products are hidden in the archive to users who do not have PI rights, or who have not been designated as co-Users of the data by the the PI: other users only see an icon warning that the data is not yet viewable. When public, any registered Herschel user can retrieve and publish the data and any person can see the browse products, where available, through the Internet.

Some additional data will be made public in the future. Many calibration observations (see Section 5.3) were taken by instrument teams in standard science modes and can be used for science. The use of standard science modes to take calibration data started even before the first scheduled science observations were made. Definitive lists of releasable calibration observation for each instrument are being prepared as of late 2013. These data will be made available in the archive when the final contents of the release are agreed by all parties.