Chandra Flight Operations Pandemic Response

Have you been working from home the past few months? Most of the Center for Astrophysics | Harvard and Smithsonian staff has, too—but not everyone! To ensure that our critical research continues, some essential Chandra X-ray Observatory employees have safely continued their work on-site at our Operations Control Center in Burlington, MA. These employees keep the Chandra spacecraft operating and orbiting safely.

Recently, our Operations Control staff answered questions posed by our followers on Twitter and Facebook, including what it's like operating a NASA Mission Control center in the midst of a pandemic -- and shared an account of some unexpected excitement!

Q&A with Flight Director Jan Vrtilek

Jan Vrtilek has been a member of Chandra's Science Mission Planning team since before launch, and has more recently become one of Chandra's three Flight Directors. His scientific interests center on the study of groups and clusters of galaxies.

Chandra was launched on July 23, 1999, and has accordingly been operating for over 20 years now. What was its original intended operating life?

Chandra was designed to operate for five years and was required to meet its Level I requirements (specifications needed to achieve its most important science goals) for only three years. Although there are a few slightly ambiguous fine points, Chandra arguably still meets these requirements now!

How is Chandra controlled and maintained from the ground, and how does the science data taken on orbit reach the scientists who analyze them?

Unlike spacecraft in low Earth orbit, which could be reached by the space shuttle during its years of operation and potentially serviced like the Hubble Space Telescope, Chandra is in a high orbit (an ellipse that gets within a few thousand kilometers of the Earth at the closest point and about halfway to the Moon at its farthest). This orbit provides a number of advantages, including very high operating efficiency, but means that our only contact with the spacecraft is by radio communication. This communication is provided through NASA’s Deep Space Network, a set of antennas and supporting equipment located in California, Spain, and Australia. Approximately three times a day, for about an hour each time, the Chandra Operations Control Center (OCC) has realtime contact with the spacecraft, which we use to verify Chandra’s overall state of health, downlink science instrument and engineering data obtained since the last contact, and uplink commands for upcoming science observations and management of the spacecraft. Additionally, software for the numerous spacecraft functions under onboard computer control can be remotely upgraded to compensate for changes in the spacecraft and to extend capabilities.

The downlinked data, once arrived at the OCC and checked for completeness, goes to multiple processing streams. One makes telemetry available to the engineering and science instrument teams for monitoring; another prepares the science data in standard formats for analysis. Validated science data are then made available, typically within about 24 hours from the end of the observation onboard the spacecraft, by network transfer to the scientists who initiated the observations through their successful proposals.

Suppose I’m a scientist who wants to use Chandra. How would I go about this? Has this changed during the pandemic?

Chandra is available by proposal to scientists all over the globe. Annually the Chandra X-ray Center issues a Call for Proposals that requests innovative and scientifically well-justified ideas for the use of Chandra; the resulting proposals are peer-reviewed by panels of subject experts. The surviving proposals -- only a fraction of those submitted, as there is always much more demand for Chandra observations than can be met -- then pass through a process of checking, refinement, and scheduling that finally results in obtaining the science observations.

This process has barely changed during the pandemic. As described elsewhere in this article, the spacecraft operations are conducted with minimal on-site staff, and the peer-review process, which used to be held in-person with panel members gathered in meeting rooms, will now be online. But these changes are behind-the-scenes for proposing scientists, who should see no difference from the pandemic in the data provided by the spacecraft and the supporting teams. Additionally, Chandra is being operated at its full pre-pandemic efficiency, so that data is flowing to the science community with all the accustomed quality, amount, and punctuality.

Q&A with Flight Operations, Mike Leger

Mike joined the Northrop Grumman Chandra Flight Operations Team in 2010 as a Mission Planner, bringing with him experience from working operations for the Space Shuttle Orbiter. He has since transitioned into the role of Command Controller, in which he is part of a two person team responsible for all real-time spacecraft commanding and first line monitoring of Chandra's health and safety and response to anomalous conditions.

Chandra is run from a custom-designed facility in Burlington. Can you tell us some of what's special about this facility and its capabilities?

Our facility in Burlington is known to the team as the Operations Control Center or "OCC." The OCC houses specialized computer systems that are isolated from the outside world (no internet) to allow the team to safely create instructions for the satellite, send those instructions to Chandra and also to command and monitor Chandra when we have live communications windows with the satellite. The OCC is also a testing platform that allows engineers to test software updates for Chandra using actual satellite hardware here on the ground. The testing platform allows controllers and engineers to practice running procedures and simulating problems and emergencies so that the team is prepared when things don’t quite go according to plan.

Can you describe your role supporting Chandra? Has it changed with the COVID-19 response actions?

My role specifically as a controller has not changed dramatically except that my commute has seen a dramatic drop in traffic for the last few months. In the office, many of the tasks that would normally be distributed over a larger team of planners and engineers now can only be completed by those of us allowed in the building. In addition to our normal tasking, many of us have taken on these added roles to help keep things running smoothly. This has required a sharper attention to emails, phone calls and timelines in order to keep products flowing and allow team members working from home to continue to support the mission with as little loss to efficiency as possible. Tasks that used to be performed by other team members such as building command load instruction sets or generating data using certain on-site tools such as Satellite Tool Kit are tasks that I now help handle with the OCC tools and systems. The results are then securely transferred to our teammates who are continuing to maintain social distancing protocols where they can finish the detailed analyses. This team process has helped us to maintain the safety of our team members while still being able to provide the high quality support and products expected by SAO safely, on time and error free.

What is it like to be one of the few Chandra team members continuing to work at the control center during the COVID-19 response?

Being one of the few team members continuing to work at the OCC during the CoVID-19 response has been both a source of pride and a personal challenge on multiple fronts. Trying to balance working a constantly rotating day/night schedule with home and family life concerns has been very hard. Prior to working on the Chandra Team I worked on the Space Shuttle Launch Team. While there, being part of a “Rideout Team” for things such as hurricanes or participating in operations that were deemed hazardous or particularly arduous was considered a badge of honor. I try to bring that mindset to this response as well as we consider ourselves the "Chandra Flight Operations Pandemic Rideout Team." In many cases the pride of being part of such a select crew working to combat an emergency that has never been encountered by the space program along with my experience and training in the emergency medical services and firefighting has helped to combat the added stresses I have encountered.

Chandra Safe Mode Response During Pandemic with Scott Wolk

Outside of day-to-day operations, on rare occasions the Chandra Operations Control staff must put their training to the test when the spacecraft doesn't operate quite as it should. When this happens, the satellite goes into Safe Mode. Flight Director Scott Wolk describes how such an event was recently managed in the midst of the current pandemic.

Scott Wolk has been a member of the Science Operations Team since before Chandra's launch. His responsibilities have included monitoring and trending spacecraft systems as well as coordinating communication with Chandra observers. He is currently one of Chandra's three Flight Directors. His scientific interests center on the study of star formation and extra-solar planets.

On May 24, 2020 at 10:17 am EDT, Chandra Spacecraft entered a Safe Mode event. Safe Mode was detected by controllers during a scheduled contact with the spacecraft which started at 5:10 pm EDT. In this case, it was due to an unintended command load interaction in the onboard software between re-enabling the Pointing Control and Aspect Determination (PCAD) system momentum monitor after a grating move and the start of the maneuver sequence. When the PCAD system transition was delayed by 4 seconds, it tripped the spacecraft into Safe Mode, and staff determined that no hardware issues had been identified as contributing to Safe Mode entry. By May 27, the team had returned the spacecraft to normal operations.

Under normal working circumstances, the Chandra team would have traveled to the Operations Control Center in Burlington, MA to work through the issues, however, given the pandemic, everything needed to be done remotely. This had never happened before! Staff all over New England jumped on to video calls to work through what had happened and to determine whether the spacecraft was safe. Luckily, the Chandra team had developed pre-existing tools that aided in returning the spacecraft to normal operating mode and allowed the team to rapidly troubleshoot at all hours of the day.