services

The services module provides a easy way to interact with the well-known services:

Services and corresponding classes

Online services	classes	Description
HyP3	HyP3Service	a class to manage HyP3 user information and jobs
	InSARMission	a class to manage HyP3 INSAR_GAMMA jobs
	InSARBurstMission	a class to manage HyP3 INSAR_ISCE_BURST jobs
LiCSAR	LiCSARService	a class to retrieve LiCSAR data
Sentinel-1 Orbit	SentinelOrbit	a class to retrieve Sentinel-1 orbit data links.

Auxiliary classes for the HyP3

classes	Description
Jobs	A class to manage HyP3 jobs. This class is designed to be used as the jobs attribute of the HyP3Service class
JOB_TYPE	A class containing the job types that can be submitted to HyP3
STATUS_CODE	A class containing the status of jobs that have be submitted to HyP3

Online Services

HyP3

class data_downloader.services.HyP3Service(username: str | None = None, password: str | None = None, prompt: bool = False, include_expired=False)

Bases: object

Class to manage HyP3 user information and jobs

__init__(username: str | None = None, password: str | None = None, prompt: bool = False, include_expired=False)

Initialize the HyP3Service class

Parameters:

• username (str, optional) – Username and password for HyP3

• password (str, optional) – Username and password for HyP3

• prompt (bool, optional) – Prompt for the username and password in the terminal, by default False.

• include_expired (bool, optional) – Include expired jobs, by default False

flush_jobs()

Flush the jobs

flush_info()

Flush the user’s information

flush()

Flush the jobs and the user’s information

login(username: str | None = None, password: str | None = None, prompt: bool | None = False)

Login to HyP3. If neither username/password nor prompt is provided, it will attempts to use credentials from a .netrc file. If prompt is True, the username and password will be prompted in the terminal. Otherwise, the username and password must be provided.

Note

This method will be called automatically when the class is initialized. However, you can change the user by calling this method again with the new username and password.

Parameters:

• username (str, optional) – Username and password for HyP3

• password (str, optional) – Username and password for HyP3

• prompt (bool, optional) – Prompt for the username and password in the terminal, by default False.

property my_info

user’s information

property jobs: Jobs

all jobs (not expired by default, set include_expired=True to include expired jobs)

class data_downloader.services.InSARMission(service: HyP3Service, granules: Series, job_parameters: dict = {})

Bases: object

Class to manage INSAR_GAMMA jobs, which are the interferograms generated by GAMMA. This class provides a pythonic interface to submit and download INSAR_GAMMA jobs from HyP3.

job_type = 'INSAR_GAMMA'

date_idx = 5

__init__(service: HyP3Service, granules: Series, job_parameters: dict = {}) → None

Initialize the InSARJob class

Parameters:

• service (HyP3Service) – HyP3Service instance to submit the job and check the submitted jobs.

• granules (pd.Series) – A pandas Series containing the granules information. The index of the Series should be the date of the granule, and the values should be the granule name.

• job_parameters (dict, optional) – Arguments to be passed to the job, by default {}. You can also change the job parameters after initializing the class by job_parameters attribute.

property jobs_on_service: Jobs

property job_parameters: dict

Job parameters

property pairs_succeed: Pairs

Pairs that succeeded in the job submission

property pairs_failed: Pairs

Pairs that failed in the job submission

jobs_to_pairs(jobs: Jobs) → Pairs | None

submit_jobs(pairs: Pairs, skip_existing: bool = True)

Submit the job to HyP3

Parameters:

• pairs (Pairs) – Pairs to be submitted to HyP3

• skip_existing (bool, optional) – Whether to skip the existing pairs that have succeeded or are running, by default True

download_jobs(output_dir: Path | str, name: str | None = None, request_time: datetime | str | slice | None = None, unzip: bool = True, remove_zip: bool = True, overwrite: bool = False, wait_running=True, wait_minutes=60, retry=3)

class data_downloader.services.InSARBurstMission(service: HyP3Service, granules: Series, job_parameters: dict = {})

Bases: InSARMission

Class to manage INSAR_ISCE_BURST jobs, which are the burst interferograms generated by ISCE. This class provides a pythonic interface to submit and download INSAR_ISCE_BURST jobs from HyP3.

job_type = 'INSAR_ISCE_BURST'

date_idx = 3

__init__(service: HyP3Service, granules: Series, job_parameters: dict = {}) → None

Initialize the InSARJob class

Parameters:

• service (HyP3Service) – HyP3Service instance to submit the job and check the submitted jobs.

• granules (pd.Series) – A pandas Series containing the granules information. The index of the Series should be the date of the granule, and the values should be the granule name.

• job_parameters (dict, optional) – Arguments to be passed to the job, by default {}. You can also change the job parameters after initializing the class by job_parameters attribute.

download_jobs(output_dir: Path | str, name: str | None = None, request_time: datetime | str | slice | None = None, unzip: bool = True, remove_zip: bool = True, overwrite: bool = False, wait_running=True, wait_minutes=60, retry=3)

property job_parameters: dict

Job parameters

property jobs_on_service: Jobs

jobs_to_pairs(jobs: Jobs) → Pairs | None

property pairs_failed: Pairs

Pairs that failed in the job submission

property pairs_succeed: Pairs

Pairs that succeeded in the job submission

submit_jobs(pairs: Pairs, skip_existing: bool = True)

Submit the job to HyP3

Parameters:

• pairs (Pairs) – Pairs to be submitted to HyP3

• skip_existing (bool, optional) – Whether to skip the existing pairs that have succeeded or are running, by default True

LiCSAR

class data_downloader.services.LiCSARService(frame_id: str, root_url: str = 'https://gws-access.jasmin.ac.uk/public/nceo_geohazards/LiCSAR_products')

Bases: object

a class to retrieve LiCSAR data

Example:

>>> from pathlib import Path
>>> import pandas as pd
>>> from data_downloader import downloader, services
>>> # specify the folder to save data
>>> home_dir = Path("/Volumes/Data/LiCSAR/106D_05248_131313/")
>>> pair_dir = home_dir / "GEOC"

init LiCSARService by frame id and download all metadata files

>>> licsar = services.LiCSARService("106D_05248_131313")
>>> downloader.download_datas(licsar.meta_urls, folder=home_dir, desc="Metadata")

generate mask data by primary_dates, secondary_dates and day span

>>> pairs = licsar.pairs
>>> mask = (pairs.primary>pd.to_datetime("2019-01-01")) & (pairs.primary<pd.to_datetime("2019-12-31")) & (pairs.days < 12 * 5 + 1)

download interferograms and coherence files filtered by mask

>>> downloader.download_datas(licsar.ifg_urls[mask].values, folder=pair_dir, desc="Interferogram")
>>> downloader.download_datas(licsar.coh_urls[mask], folder=pair_dir, desc="Coherence")

__init__(frame_id: str, root_url: str = 'https://gws-access.jasmin.ac.uk/public/nceo_geohazards/LiCSAR_products')

init LiCSARService.

Parameters:

frame_idstr

frame id of LiCSAR.

root_urlstr

root url of LiCSAR. default is “https://gws-access.jasmin.ac.uk/public/nceo_geohazards/LiCSAR_products”.

property pairs: Pairs

all available pairs of given frame id.

property ifg_urls: Series

interferogram urls of given frame id.

property coh_urls: Series

coherence urls of given frame id.

property urls: DataFrame

all urls, including interferogram and coherence, of given frame id.

property meta_urls: ndarray

metadata urls of LiCSAR.

metadata includes:

• E, N, U, hgt

• baselines

• metadata.txt

• network.png

• frame_id-poly.txt

Sentinel-1 Orbit

class data_downloader.services.SentinelOrbit(home_aux_cal: str = 'https://s1qc.asf.alaska.edu/aux_cal/', home_preorb: str = 'https://s1qc.asf.alaska.edu/aux_poeorb/')

Bases: object

a class to retrieve Sentinel-1 orbit data links.

Examples

>>> from data_downloader import downloader, services
>>> from pathlib import Path

>>> folder_cal = Path("/media/data/aux_cal")  # specify the folder to save aux_cal
>>> folder_preorb = Path("/media/data/poeorb")  # specify the folder to save aux_poeorb

init SentinelOrbit:

>>> s1_orbit = services.SentinelOrbit()

Get all aux_cal data links of S1A and S1B and download them:

>>> urls_cal = s1_orbit.cal_urls(platform='all')
>>> downloader.async_download_datas(urls_cal, folder=folder_cal)

Get all precise orbit data links of S1A during 20210101-20220301 and download them:

>>> urls_preorb = s1_orbit.poeorb_urls(
>>>    date_start="20210101", date_end="20220301", platform="S1A"
>>> )
>>> downloader.download_datas(urls_preorb, folder=folder_preorb)

__init__(home_aux_cal: str = 'https://s1qc.asf.alaska.edu/aux_cal/', home_preorb: str = 'https://s1qc.asf.alaska.edu/aux_poeorb/') → None

init SentinelOrbit.

Parameters:

• home_aux_cal (str) – home url of aux_cal, default is “https://s1qc.asf.alaska.edu/aux_cal/”.

• home_preorb (str) – home url of aux_poeorb, default is “https://s1qc.asf.alaska.edu/aux_poeorb/”.

cal_urls(platform: Literal['S1A', 'S1B', 'all'] = 'all')

filter files from urls of aux_cal by platform.

Parameters:

platform (str, one of ['S1A', 'S1B','all']) – platform of satellite. should be one of [‘S1A’, ‘S1B’,’all’]

poeorb_urls(date_start: str, date_end: str, platform: Literal['S1A', 'S1B', 'all'] = 'all')

filter files from urls of aux_poeorb (precise orbit) by date and platform.

Parameters:

• date_start (str) – start/end date to filter, can be any format that can be converted by pd.to_datetime (e.g. ‘20210101’, ‘2021-01-01’, ‘2021/01/01’).

• date_end (str) – start/end date to filter, can be any format that can be converted by pd.to_datetime (e.g. ‘20210101’, ‘2021-01-01’, ‘2021/01/01’).

• platform (str, one of ['S1A', 'S1B','all']) – platform of satellite. should be one of [‘S1A’, ‘S1B’,’all’]

Auxiliary classes for the HyP3

Jobs

class data_downloader.services.hyp3.Jobs(jobs: list[Job])

Bases: object

A class to manage HyP3 jobs. It provides a pythonic interface to filter and select jobs using the numpy and pandas libraries. This class is designed to be used as the jobs attribute of the HyP3Service class.

__init__(jobs: list[Job]) → None

Initialize the Jobs class

Parameters:

jobs (list[sdk.Job]) – List of Job objects from HyP3 SDK. You can get the jobs from the hyp3_sdk.Batch.jobs attribute.

property job_type: ndarray

the job type of all jobs

property job_id: ndarray

the job ID of all jobs

property request_time: ndarray

the request time of all jobs

property status_code: ndarray

the status code of all jobs

property user_id: ndarray

the user ID of all jobs

property name: ndarray

the name of all jobs

property job_parameters: ndarray

the job parameters of all jobs

property files: ndarray

the files of all jobs

property file_names: ndarray

the file names of all jobs

property file_urls: ndarray

the file urls of all jobs

property file_sizes: ndarray

the file sizes of all jobs

property logs: ndarray

the logs of all jobs

property browse_images: ndarray

the browse images of all jobs

property thumbnail_images: ndarray

the thumbnail images of all jobs

property expiration_time: ndarray

the expiration time of all jobs

property processing_times: ndarray

the processing times of all jobs

property credit_cost: ndarray

the credit cost of all jobs

property total_credit_cost: int

the total credit cost of all jobs

property frame: DataFrame

jobs in the form of a pandas DataFrame

sel(name: str | None = None, job_type: JOB_TYPE | None = None, status_code: STATUS_CODE | None = None, request_time: datetime | str | slice | None = None) → Jobs

Select jobs based on job type and status code

Parameters:

• name (str | None) – Name of the job to filter by

• job_type (JOB_TYPE | None) – Job type to filter by

• status_code (STATUS_CODE | None) – Status code to filter by

• request_time (datetime | str | slice | None) – Request time to filter by. Can be a datetime object, a string, or a slice object. If a slice object is used, the start must be a string or a datetime object, and the stop can be None, a string, or a datetime object. If a string is used, it must be in the format that can be converted to a datetime object using pd.to_datetime. by default None

property succeeded: Jobs

all succeeded jobs (not expired by default)

property failed: Jobs

all failed jobs (not expired by default)

property pending: Jobs

all pending jobs (not expired by default)

property running: Jobs

all running jobs (not expired by default)

JOB_TYPE

class data_downloader.services.hyp3.JOB_TYPE

Bases: BaseConstants

AUTORIFT = 'AUTORIFT'

RTC_GAMMA = 'RTC_GAMMA'

INSAR_GAMMA = 'INSAR_GAMMA'

INSAR_ISCE_BURST = 'INSAR_ISCE_BURST'

__init__() → None

classmethod variables()

Returns a list of all available variables

STATUS_CODE

class data_downloader.services.hyp3.STATUS_CODE

Bases: BaseConstants

SUCCEEDED = 'SUCCEEDED'

FAILED = 'FAILED'

RUNNING = 'RUNNING'

PENDING = 'PENDING'

__init__() → None

classmethod variables()

Returns a list of all available variables