Monday, February 11, 2019

OPUS Updates to Product Downloading

We have recently added two new features to make downloading of data products easier. First, a reminder about how to use our download system:

  1. After refining your search criteria, browse the thumbnail gallery by clicking on the "Browse Results" tab.
  2. When you find an observation you want to save, click on the checkmark in the little toolbar at the bottom of the thumbnail.
  3. Once you've selected all of the observations you care about, click on the "Selections" tab.
  4. You will see the total size of the data products that will go into the zip file you can download. You will also see a list of product types that you can select or deselect to control what goes in your zip file.
  5. There are now three options for how to download data:
    1. Download CSV of Chosen Columns - This will create a single CSV file that has one row for each observation and one column for each metadata field you have selected using "Choose Columns". No actual data will be downloaded, only a list of metadata.
    2. Download Zipped Data Archive - This will create a ZIP file that you can download separately. The ZIP file contains all of the data products for the observations you chose. It also includes a file containing product checksums, a manifest, and the same CSV file discussed above. Finally, it includes a new file, urls.txt, discussed below.
    3. Download Zipped URL Archive - This will create a ZIP file that is the same as the one above, but with no actual data products. It will only include the checksums, manifest, CSV, and URLs files.
The new urls.txt file solves the problem of downloading huge amounts of data when it is impractical to put all the data into a single ZIP file. urls.txt simply contains a list of URLs for all data products for all observations you selected, using the PDS Ring-Moon Systems Node ViewMaster site as the source for the data. You can easily use this file to automatically retrieve the actual data using the wget command as follows:

wget -i urls.txt

We have also made it easier to download ZIP archives for a single observation directly from the Details tab:

As always, you can continue to download individual data products from this tab as well.

We value your feedback, questions, comments, and suggestions. Please let us know what you think!

Wednesday, January 16, 2019

OPUS Now Has Input Value Checking and Search String Hints

One of the most-requested improvements in OPUS is now available: Checking numerical inputs for validity.

As you type into any numerical field (including floating point, integer, date/time, and spacecraft clock count), the field border will change as you're typing. Green means the current value is valid as-is. Yellow means the current value is invalid. If you try to do a search with an invalid value, the erroneous search field will turn red and no search will be performed until you fix the input.

In addition, once you've entered a valid value, that value is normalized. For a floating point value, that means it is formatted using the proper number of digits to the right of the decimal point to indicate its precision. For a date/time value, that means converting it to the standard format YYYY-MM-DDTHH:MM:SS.sss.

As an added bonus, string input fields now have string hints. As you type, the database will be searched for appropriate matches and they will be presented to you in a drop-down box. If your search is reasonably constrained, these hints will only be those relevant to your current search. If your search has too many results, however, the hints will indicate all possible matches from the entire database. Up to 100 hints can be shown, and the hints intelligently obey the search type (contains, begins, ends, etc.). Simply click on a result if you wish to choose it for your search. This feature is particularly useful for fields such as Cassini Observation Name or Note that have a lot of free-form text.

As always, we look forward to hearing from our users. Please feel free to ask questions, make suggestions, or complain, as appropriate! We want everyone's experience using OPUS to be a great one.

Monday, December 10, 2018

Recent updates to OPUS (Instrument Names, Units, and Versions!)

  • Several fields (e.g. Spacecraft Clock Count, Filter) have similar names across instruments, making it difficult to tell which instrument or mission a particular search field or metadata column refers to. We now include the instrument or mission name in the search field title as well as in the metadata columns in the same way that we show the name of the body for surface geometry.

  • Many users have been requesting information on the measurement units supported by OPUS, and we are pleased to announce that these units are now shown. A future release will allow the user to select their own units.

  • The PDS archive supports multiple versions for data (like any good archive, nothing is ever thrown away). These versions are now visible in OPUS on the Details page. For now, when downloading a zip file for selected images, only the current version is returned. In a future release we will allow you to select which version you wish to download. 

  • Previously, Cassini ISS filter names lumped all polarizing filters of a particular type (e.g. IRP0, IRP90) into a single filter (IRPOL). We have now separated these out, making it easier to determine the polarization angle when looking at the metadata.
  • We now ignore spaces, commas, and underscores in numeric fields, allowing input such as "100,000" or "100_000" to indicate thousands.
  • Displayed numeric values (either the min/max hints on the search page or metadata values) are now rounded to an appropriate number of significant digits, with trailing zeros included to show the available precision.

Saturday, November 17, 2018

Recent updates to OPUS (new OPUS ID format!)

  • There is now a link for Questions and Feedback in the top menu bar. Please use it!
  • The format of the OPUS ID (used to uniquely identify observations) has changed. The previous format was incompatible with the upcoming migration to PDS4. The new OPUS ID format should stay consistent for the foreseeable future. The old OPUS ID format is no longer supported. Some examples of the change:
        Old: COISS_1001-1294561143_1295221348-W1294561143_1
        New: co-iss-w129456114
        Old: NHLALO_x001-20060224_000310-lor_0003103486_0x630
        New: nh-lorri-lor_0003103486
  • We have removed the short-lived search fields Spatial Sampling, Wavelength Sampling, and Time Sampling and returned to a simpler way of describing observations called Observation Type. Current choices include:
    • Image (a 2-D array of intensity values)
    • Spectrum (a 1-D array of intensity vs. wavelength)
    • Spectral Image (similar to Image, but a grating or prism has been used to spread out the spectrum along one of the axes)
    • Spectral Cube (a 3-D array, similar to an Image but with each pixel value containing an entire spectrum)
    • Time Series (a 1-D array of intensity or optical depth vs. time, such as for an occultation)
    • Spectral Time Series (similar to Time Series, but containing a spectrum for each time step)
  • The Primary File Spec for Galileo SSI observations has been changed. The original format, which looked like "GO_0017:[J0.OPNAV.C034640]5900R.IMG", was based on an ancient filesystem constraint that required the actual filename to be split across multiple directories because it was too long. This resulted in the final filename, such as "5900R.IMG", being non-unique. We have changed the format to have a more modern, non-restricted format such as "GO_0017/J0/OPNAV/C0346405900R.IMG", where each filename is now unique.
  • When downloading observations that have been selected, the available products are now categorized and sorted by their product type, such as "Metadata Products", "Browse Products", and "Mission-Specific Products".
  • The definition of Planet has been changed for Cassini and New Horizons instruments. Previously, Planet was based on the Intended Target Name, so that an observation of Venus was marked as planet Venus. Now, consistent with other missions and instruments, the Planet field indicates approximately where the spacecraft was when the observation was taken. For most observations, this is a good starting point when looking for images of that particular planet, but if you want to be more specific use the Intended Target Name or Surface Geometry Target Name. The exception to this is the Hubble Space Telescope, which is always around Earth. In this case, the Planet field is still based on the Intended Target Name.
  • The definition of Measurement Quantity for Cassini UVIS has been changed. The choice between Optical Depth and Emission is no longer dependent on the instrument mode, but instead depends on the textual Note included with the observation. Any observation with a Note containing "OCCULTATION" is marked as Measurement Type Optical Depth. All other observations are marked as Emission.
  • We have added descriptive text to help users understand the difference between Intended Target Name and Surface Geometry Target Name.
  • For added clarity, we have renamed some targets:
    • J Rings → Jupiter Rings
    • S Rings → Saturn Rings
    • U Rings → Uranus Rings
    • N Rings → Neptune Rings
    • J Minor Sat Jupiter Minor Satellites
    • Cal Lamps Calibration Lamps
  • It is now possible to right-click on the View Detail link anywhere it appears and open OPUS in a new browser tab or window with the Detail tab selected.
  • The /api/files JSON return has been changed. Previously the product type was identified by a "pretty name" such as "Raw Data". We have changed this to a simple slug-style name such as "coiss-raw" to decouple the user experience from the internal data representation. The precise data products available for each instrument are complicated (it's not as simple as just "raw" or "calibrated" when you look at all the things we currently support or are going to support). We decided to split them out by instrument so that we could give the full information to the user, rather than risk merging data products together that really aren't in the same category. Anything that is truly generic, such as browse images (browse-thumb, browse-small, etc.) or geometry indexes (ring-geometry, moon-geometry) do not have the instrument name in the front. Anything that could be different from instrument to instrument (e.g. coiss-raw, coiss-calibrated, or coiss-thumb [a special COISS-specific browse image - it's not the same thing as browse-thumb]) has the instrument name prepended. If you only want raw images, it should be easy enough to just look for keys ending in "raw", but be aware that some instruments, like Hubble, may have a different concept of what raw means.
  • Several annoying bugs were fixed. It is once again possible to sort in the table view on a field that contains multiple choices. There was a 3-order-of-magnitude error in the Observation Duration for certain instruments. Some HST NICMOS aperture names were listed twice, with only one working. Some observations were being treated as separate when they were really just separate products available for the same observation; fixing this results in a slightly smaller number of search results for some instruments. It is now possible to download all downlink copies of certain Cassini VIMS observations.

Thursday, August 30, 2018

New Hubble Images Added to OPUS

More than 6,000 solar system images taken by the Hubble Space Telescope have been added to OPUS. The majority of these images were taken by Hubble's Advanced Camera for Surveys (ACS) and Wide Field Camera 3 (WFC3), although there are also new images from NICMOS, STIS, and WFPC2. Most new images are of Jupiter and Saturn, but there are also new images of Uranus, Neptune, Pluto, and even Earth's Moon.

Thursday, August 23, 2018

Recent Additions to the OPUS API (/opus/api/metadata_v2)

The /opus/api/metadata/<OPUSID>.json API has long had a problem. While you request specific metadata using the "cols=" argument with slug names such as "opusid" or "SURFACEGEOtitancenterdistance", the returned data structure uses different names for those fields. This makes matching the returned values to the requested values difficult (they are at least given in the same order). On top of that, the returned names can be ambiguous. Take, for example, the query:,instrument,planet,target,time1,observationduration,SURFACEGEOsaturncenterdistance,SURFACEGEOtitancenterdistance
This returns:
[{"opus_id": "COISS_2111-1866071296_1866225122-N1866071296_1"},
 {"instrument_id": "COISS"},
 {"planet_id": "SAT"},
 {"target_name": "TITAN"},
 {"time1": "2017-048T23:59:39.059"},
 {"observation_duration": 46.0},
 {"center_distance": 272061.836},
 {"center_distance": 272061.836}
Even though we queried on, for example, "opusid", we were returned a field named "opus_id". Worse yet, we have two returned fields called "center_distance" without a way to disambiguate them (and they are incorrectly the same value).

A second problem is that some of the returned values, such as "COISS" or "SAT", are internal representations, not the values normally shown to the user such as "Cassini ISS" or "Saturn".

To fix these problems while still providing backwards compatibility, we have introduced a second version of this API:
This version fixes all of the problems mentioned above. For example, the similar query:,instrument,planet,target,time1,observationduration,SURFACEGEOsaturncenterdistance,SURFACEGEOtitancenterdistance
[{"opusid": "COISS_2111-1866071296_1866225122-N1866071296_1"},
 {"instrument": "Cassini ISS"},
 {"planet": "Saturn"},
 {"target": "Titan"},
 {"time1": "2017-048T23:59:39.059"},
 {"observationduration": 46.0},
 {"SURFACEGEOsaturncenterdistance": 1284796.671},
 {"SURFACEGEOtitancenterdistance": 272061.836}
Note that the labels are the same as the requested slugs, and the values are the same as seen in the main OPUS web UI.

We strongly suggest that you use the new version of the metadata API going forward to avoid these problems.

Cassini End-of-Mission ISS, UVIS, and VIMS Volumes (COISS_2115, COISS_2116, COUVIS_0060, COVIMS_0093, COVIMS_0094)

The final volumes from the Cassini End of Mission (July 1, 2017 through September 13, 2017) are now available in OPUS. These include COISS_2115, COISS_2116, COUVIS_0060, COVIMS_0093, and COVIMS_0094, all with enhanced geometric metadata.

See also how to interpret COUVIS preview images and how to interpret COVIMS preview images.

COCIRS is Again Available in OPUS

The PDS Ring-Moon Systems Node has a limited collection of Cassini CIRS observations available as separate, searchable data products. Due to the incomplete coverage, we removed CIRS from OPUS during our recent major upgrade. However, by popular demand, we are once again including CIRS in our search results. Please note that we only have CIRS data from June 1, 2004 through June 30, 2010. Complete coverage of CIRS will be available sometime in 2019.

See also how to interpret COCIRS preview images.

Sunday, August 5, 2018

Major OPUS Software and Data Release

We are pleased to announce a major update to both the OPUS software and the OPUS metadata.
  • All metadata has been re-imported to our database using a brand new import pipeline that dramatically improves accuracy and consistency. Much of the metadata has been improved, either through our own modifications as problems were found or through re-release of the metadata from the instrument teams.
  • Images from HST NICMOS and HST STIS have been added. All observations from Cassini CIRS have been removed pending an upcoming major effort to reformat the data. All data is up-to-date through the Q1'2018 release. We will add the Q2 (including Cassini end-of-mission) data shortly once it becomes available in our main repository. 
  • The layout of the OPUS search fields has changed somewhat, and some fields have been added or removed.
    • The original General Constraints has been split into General Constraints, which contains data about the mission, instrument, and targets, and PDS Constraints, which contains information only relevant to the internal workings of the PDS (volume ID, product creation time, etc.). For General Constraints:
      • Planet now contains Other for searching on observations that do not directly apply to a particular planet, and Intended Target Name includes Other as well, providing access to KBOs, asteroids, stars, etc. Earth has also been added, allowing searches on Earth as well as Earth's Moon.
      • Nominal Target Class has been expanded to include both Regular Satellite and Irregular Satellite.
      • Data Type has been removed and replaced with three new search fields: Spatial Sampling (Point, 1-D, 2-D), Wavelength Sampling (YesNo), and Time Sampling (Yes, No). For example, a COISS or VGISS image has 2-D spatial sampling, no wavelength sampling, and no time sampling. A COUVIS occultation has 1-D spatial sampling, no wavelength sampling, and time sampling. An HST image with a prism or grism has 2-D spatial sampling, wavelength sampling, and no time sampling.
    • For PDS Constraints:
      • There are new search fields Data Set ID, Product ID, and Product Creation Time.
    • The Image Constraints menu is now always available, making it easier to search on image attributes across all instruments, ignoring instruments that do not actually produce images.
    • The Surface Geometry Constraints and Ring Geometry Constraints menus contain the same information as before, but have been reorganized to be consistent with each other.
    • The Cassini Mission Constraints menu has new search fields Mission Phase, Cassini Target Code, and Sequence ID. It also now has the ability to search on Saturn Orbit Number either by numerical range or by checkboxes. Note that the numerical range understands the oddly named orbits 00A, 00B, and 00C.
    • The Cassini ISS Constraints menu has new search fields Instrument Mode, Target Description, and Image Number. The latter is particularly useful for searching by a range of image numbers without regard to the "N" or "W" prefix. Fields related to the inner workings of compression and telemetry have been removed.
    • The Cassini UVIS Constraints menu no longer has the High Voltage Power Supply State or ODC ID fields.
    • The Cassini VIMS Constraints menu has new search fields Swath Width, Swath Length, and Star Tracking.
    • The Galileo SSI Constraints menu no longer has the Data Set ID (moved to the PDS menu), Orbit Number (duplicate with Galileo Mission menu), Encoding Compression Ratio, or Processing History Text fields.
    • The Voyager Mission Constraints menu now has the Mission Phase search field, but has lost the Spacecraft Name field (which is redundant with Instrument Host Name in the General Constrains menu).
    • The Voyager ISS Constraints menu has new search fields Usable Lines and Usable Samples.
    • The Hubble Mission Constaints menu has new search fields Filter Type (Long Pass, Ramp, etc.), WFPC2 PC1/WF2/WF3/WF4 Flag, and Publication Date. The Hubble filter names are sorted by instrument, which results in a large number of checkboxes. We recommend some combination of searching by filter type and wavelength instead.
    • For consistency, there is a new New Horizons Constraints menu, which includes Mission Phase and Spacecraft Clock Count.
    • The New Horizons LORRI Constraints menu has lost the Nominal Target Center Resolution, Phase Angle, and Target Center Distance fields, as these are redundant with the superior information available in the surface geometry metadata. Likewise the New Horizons MVIC Constraints menu has lost the Phase Angle and Target Center Distance fields.
  • It is now possible to search on ranges of Spacecraft Clock Count in the encoded format (e.g. "08966:30:752").
  • We have attempted to keep the public API backwards compatible, while also improving it. There is a fleshed-out version of the API Guide available by clicking on "API" in the footer of the main OPUS page. This page will continue to be improved; suggestions are welcome.
  • The old field "ringobsid" is supported for backwards compatibility, but will not be available for new observations going forward. The new field is called "opusid" and is in a completely different format. We strongly recommend that you switch to using opusid as soon as possible and remove all references to ringobsid from your code.
  • There has been a major renaming of the "slugs" (URL elements) used for searching. This is things like "planet=Saturn&target=Pan&CASSINIrevno=202". The previous slugs were inconsistently named and had the potential for conflicts across missions and instruments. The new names include a prefix (CASSINI, COISS, SURFACEGEO) to keep the namespaces independent. At the bottom of the API Guide there is now a table that shows all available search terms sorted by where they show up in the OPUS web UI, along with the new and old (if different) slug names. Please start using the new slug names ASAP. 
  • A large number of bugs in the UI have been fixed, and the code has been reorganized to create a solid foundation for the future improvements we have planned. If you find any problems with content or usability or just have a request for a new feature, we strongly request that you drop us a note using the "Questions and Comments" link found at the bottom of the OPUS search page, or the "Help us improve OPUS" link found in the right column of this blog. You may also submit issue reports directly to our GitHub repository.

Thank you for using OPUS!