Selecting your data of interest with OSI SAF Opensearch Web Service

  • The power of this OpenSearch service relies in automating a whole sequence within a script that chains the granule search, the parsing of the XML result and the download of the data files.

  • Polar orbiting satellites such as the Metop series provide daily high resolution observations of sea surface temperature (SST) from AVHRR instrument along the satellite track. In order to reduce the size of the AVHRR data files and to allow users to download only the data files covering their area of interest, the AVHRR swath data (Level 2P products: OSI-204-b from Metop-B, OSI-204-c from Metop-C) are broken down into smaller granules covering 3 minutes of continuous acquisition. Hundreds of such granules are made available every day, covering the whole earth, as illustrated on figure 1 which displays the location of these granules over 12 hours.

    • Location and reference number of each granule of sea surface temperature from AVHRR for Metop-C over half a day.
  • The figure above gives location and reference number of each granule of sea surface temperature from AVHRR for Metop-C over half a day.

     

    Each of these granules is referenced with an identifier that is part of its filename: until now, users had to select their granules of interest (defined by a custom area and time frame) using these identifiers to download from OSI SAF FTP or HTTPS server only the relevant data files.

  • OSI SAF now offers a REST-based Web Service (https://opensearch.ifremer.fr) developed and operated by Ifremer for users to select the URIs of the data files matching the user’s spatial and temporal criteria.

  • This service is based on OpenSearch, a RESTful protocol that provides a way to access identified or located results, to publish search results in a standard and accessible format and to download them. OpenSearch is the protocol selected by the Group for High Resolution Sea Surface Temperature (GHRSST) to provide a single way to search worldwide for sea surface temperature products from any data provider in the next two years, and it is also offered by other data services such as EUMETSAT CODA. Opensearch queries can be handled by a large set of client tools as simple as common web browsers, download-managers or computer programs such as cURL or wget. They can then be used to build URI that the user can give in input to scripts in order to perform batch data download.

  • A URL to query OSI SAF products from our OpenSearch end-point is simply built and will look like this:

    https://opensearch.ifremer.fr/granules.atom?datasetId=<dataset_identifier>&startPage=<page_start_value>&count=<page_size>&timeStart=<start_time>&timeEnd=<end_time>&geoBox=<area>

  • The main search parameters of a query, separated by &, are:

    • the identifier of the products to search granule from (for instance avhrr_sst_metop_c-osisaf-l2p-v1.0 for Metop-C L2P granules)

    • the start and end time of the search temporal interval, respectively in timeStart and timeEnd, expressed in ISO format as YYYY-MM-DDThh:mm:ss (ex: 2021-10-01T12:00:00)

    • the geographical area, in geoBox, expressed as <longitude min, latitude min, longitude max, latitude max> (example: -6.0,30.0,36.0,46.0 to select the granules intersecting the Mediterranean Sea)

    An OpenSearch query is paginated, meaning it only returns a fraction of the results in a single call; the remaining results can be queried by requesting the next pages of results for the same query, completed with the pagination options. The available options are:

    • count=<page_size> , where page_size is the number of results listed per page

    • startPage=<page_start_value>, where page_start_value identifies from which result position starts the pagination.

    If nothing is specified by default page_size and page_start_value are set to count=1000 and startPage=0

    The OSI SAF products currently searchable through this service include all the global SST swath products for Metop series (other geostationary and lower resolution gridded SST products will be added soon):

     

    dataset_identifier

    OSI SAF dataset

    avhrr_sst_metop_a-osisaf-l2p-v1.0

    OSI-204 (Full Resolution Sea Surface Temperature from Metop-A)

    avhrr_sst_metop_b-osisaf-l2p-v1.0

    OSI-204-b (Full Resolution Sea Surface Temperature from Metop-B)

    avhrr_sst_metop_c-osisaf-l2p-v1.0

    OSI-204-c (Full Resolution Sea Surface Temperature from Metop-C)

     

  • Here is an example of such query, searching for Metop-C SST granules on the 1st october 2021 over the Mediterranean Sea:

    https://opensearch.ifremer.fr/granules.atom?datasetId=avhrr_sst_metop_c-osisaf-l2p-v1.0&startPage=0&count=1000&timeStart=2021-10-01T00:00:00Z&timeEnd=2021-10-01T23:59:59Z&geoBox=-6.0,30.0,36.0,46.0

  • This query can be directly pasted in a web browser. As you will see, the result in XML format is quite verbuous ! it contains all the available URLs for each found granule (in an entry XML block), as in the example below:

    <entry>
    <
    title>20211001201303-OSISAF-L2P_GHRSST-SSTsubskin-AVHRR_SST_METOP_C-sstmgr_metop03_20211001_201303-v02.0-fv01.0.nc</title>
    <
    id>https://opensearch.ifremer.fr/granules.atom?uid=avhrr_sst_metop_c-osisaf-l2p-v1.0:20211001201303-OSISAF-L2P_GHRSST-SSTsubskin-AVHRR_SST_METOP_C-sstmgr_metop03_20211001_201303-v02.0-fv01.0.nc</id>
    <
    updated>2021-10-22T11:50:21Z</updated>
    <
    dc:date>2021-10-01T20:13:03Z/2021-10-01T20:16:03Z</dc:date>
    <
    georss:box>-6.0 40.37085818544712 17.817588418080003 46.0</georss:box>
    <
    georss:polygon>-6.0 46.0 17.817588418080003 46.0 16.16071698566514 45.86873857351433 10.69122302886527 44.95585468734553 5.445601848878985 43.79644335704491 0.01140875983033872 42.2543263703058 -5.480691645732092 40.37085818544712 -6.0 41.08925799444315 -6.0 46.0</georss:polygon>
    <
    link title="DATARMOR GPFS" rel="enclosure" type="application/octet-stream" href="/home/ref-osisaf-public/data/sst/l2p/global/avhrr_metop_c/2021/274/20211001201303-OSISAF-L2P_GHRSST-SSTsubskin-AVHRR_SST_METOP_C-sstmgr_metop03_20211001_201303-v02.0-fv01.0.nc"/>
    <
    link title="FTP" rel="enclosure" type="application/octet-stream" href="ftp://eftp1.ifremer.fr/cersat-rt/project/osi-saf/data/sst/l2p/global/avhrr_metop_c/2021/274/20211001201303-OSISAF-L2P_GHRSST-SSTsubskin-AVHRR_SST_METOP_C-sstmgr_metop03_20211001_201303-v02.0-fv01.0.nc"/>
    <
    link title="HTTPS" rel="enclosure" type="application/octet-stream" href="https://osi-saf.ifremer.fr/sst/l2p/global/avhrr_metop_c/2021/274/20211001201303-OSISAF-L2P_GHRSST-SSTsubskin-AVHRR_SST_METOP_C-sstmgr_metop03_20211001_201303-v02.0-fv01.0.nc"/>
    <
    summary type="text">Granule metadata for 20211001201303-OSISAF-L2P_GHRSST-SSTsubskin-AVHRR_SST_METOP_C-sstmgr_metop03_20211001_201303-v02.0-fv01.0.nc</summary>
    </
    entry>

  • For this granule, you get in particular the HTTPS or FTP download URL. Important: OSI SAF HTTPS and FTP servers for SST products is protected, so you still need to be registered on OSI SAF web site (https://osi-saf.eumetsat.int) to get the access credentials. Having your login and password, you can then paste one of above URL in a download tool such cURL or wget :

    wget -np --http-user=login --http-password=password https://osi-saf.ifremer.fr/sst/l2p/global/avhrr_metop_c/2021/274/20211001201303-OSISAF-L2P_GHRSST-SSTsubskin-AVHRR_SST_METOP_C-sstmgr_metop03_20211001_201303-v02.0-fv01.0.nc

  • Of course the power of this OpenSearch services relies in automating this whole sequence within a script that chains the granule search, the parsing of the XML result and the download of the data files.

     

    Many libraries exist in various languages to help you parse the XML result of a query. We provide also an example python script (https://opensearch.ifremer.fr/docs/0.1/examples/granules/). You may check out also the eo-harvest python package (https://gitlab.ifremer.fr/naiad/eo_harvest) that eases the query and parsing sequence for various flavours of similar Opensearch query services, including the Ifremer one for OSI SAF products.

  • Further readings:

New ticket helpdesk