HyP3 InSAR Job Submission and Download Tutorial#
HyP3 processes whole frame interferograms using GAMMA, while it processes burst data using ISCE2. The job_type for GAMMA and ISCE2 is “INSAR_GAMMA” and “INSAR_ISCE_BURST” respectively. We design classed InSARMission and InSARBurstMission to handle the GAMMA and ISCE2 jobs respectively. Those two classes are shared same API, so you can use them in the same way.
This notebook demonstrates how to submit and download HyP3 “INSAR_GAMMA” jobs using the DataDownloader packages. You can replace InSARMission with InSARBurstMission to submit and download “INSAR_ISCE_BURST” jobs.
Prepare the data#
We encourage you to get the granule names from ASF Data Search Vertex and download the metadata in the .geojson format. This will allow you to use the DataDownloader to download the data.
load the metadata from the downloaded *.geojson files using geopandas
import pandas as pd
import geopandas as gpd
from data_downloader import services
from data_downloader import downloader as dl
asf_file = "/Volumes/Data/asf-datapool-results-2024-03-29_11-24-18.geojson"
df_asf = gpd.read_file(asf_file)
df_asf = df_asf[df_asf.processingLevel == "SLC"] # only use SLC
create a pandas Series where the granule names are the values, and the dates of the granules are used as the index.
# dates format in df_asf are mixed, convert them to standard date format
dates_str = (
pd.to_datetime(df_asf.startTime, format="mixed")
.map(lambda x: x.strftime("%F"))
.values
)
dates = pd.to_datetime(dates_str)
# create a series with dates as index and sceneName as values
granules = pd.Series(df_asf.sceneName.values, index=dates)
To generate pairs to be submitted, you can use the PairsFactory class in the FanInSAR module.
# generate pairs using given dates
pairs_factory = fis.PairsFactory(dates)
pairs = pairs_factory.from_interval(max_interval=3, max_day=180)
Submit jobs#
To begin, we need to initialize the HyP3Service and InSARMission classes. The HyP3Service class handles the login and retrieval of user information, while the InSARMission class is responsible for submitting the INSAR_GAMMA jobs.
service = services.HyP3Service()
mission = services.InSARMission(granules=granules, service=service)
Since the look_vectors, dem, and inc_map data are almost the same for all interferograms, we can submit them only once and then ignore them for subsequent submissions. This approach will significantly save storage space
# submit first job to get shared data (DEM, incidence angle map, etc.)
args = {
"name": "RGV_2024_03_first", # give a unique name to find this job later
"include_look_vectors": True,
"include_dem": True,
"include_inc_map": True,
"looks": "10x2",
"phase_filter_parameter": 0.5,
}
mission.job_parameters = args
mission.submit_jobs(pairs[:1], skip_existing=True)
# submit the rest of the jobs without shared data (save time and storage for downloading)
args = {
"name": "RGV_2024_03",
"looks": "10x2",
"phase_filter_parameter": 0.5,
}
mission.job_parameters = args
mission.submit_jobs(pairs, skip_existing=True)
Check the status of missions
service.flush()
service
HyP3Service(
user_id=fanchengyan1995,
remaining_credits=10,
succeeded=976,
failed=0,
pending=557,
running=2
)
Submit remaining jobs using a new account#
Since one account can only submit a limited number of jobs, we can use a new account to submit the remaining jobs. To do this, we need to retrieve the pairs that have already been submitted and then submit the remaining pairs using the new account.
Note
Please note that submitted jobs may be failed and need to be resubmitted in the future, as pending jobs may fail.
pairs_submitted = mission.jobs_to_pairs(
mission.jobs_on_service.succeeded
+ mission.jobs_on_service.running
+ mission.jobs_on_service.pending
)
pairs_submitted
primary secondary
0 2014-10-10 2014-11-03
1 2014-10-10 2014-11-15
2 2014-10-10 2014-11-27
3 2014-10-10 2014-12-09
4 2014-11-03 2014-11-15
.. ... ...
877 2020-06-28 2020-07-22
878 2020-07-04 2020-07-10
879 2020-07-04 2020-07-16
880 2020-07-04 2020-07-22
881 2020-07-04 2020-07-28
[882 rows x 2 columns]
pairs_remain = pairs - pairs_submitted
pairs_remain
primary secondary
0 2015-09-11 2015-09-23
1 2015-09-11 2015-10-05
2 2015-09-11 2015-10-17
3 2015-09-23 2015-10-05
4 2015-09-23 2015-10-17
.. ... ...
618 2024-02-20 2024-03-15
619 2024-02-20 2024-03-27
620 2024-03-03 2024-03-15
621 2024-03-03 2024-03-27
622 2024-03-15 2024-03-27
[623 rows x 2 columns]
To avoid conflicts with user information stored in the .netrc file, you can log in to a new account for the HyP3 Service using the prompt.
service1 = services.HyP3Service(prompt=True)
service1
HyP3Service(
user_id=fanchengyan,
remaining_credits=10000,
succeeded=0,
failed=0,
pending=0,
running=0
)
args = {
"name": "RGV_2024_03",
"looks": "10x2",
"phase_filter_parameter": 0.5,
}
mission1 = services.InSARMission(
granules=granules, service=service1, job_parameters=args
)
mission1.submit_jobs(pairs_remain)
Multiple granules found for pair 20160414_20160520: ['S1A_IW_SLC__1SDV_20160520T053510_20160520T053537_011338_01134D_62C2', 'S1A_IW_SLC__1SDV_20160520T053446_20160520T053513_011338_01134D_68E6'].First one will be used.
Multiple granules found for pair 20160426_20160520: ['S1A_IW_SLC__1SDV_20160520T053510_20160520T053537_011338_01134D_62C2', 'S1A_IW_SLC__1SDV_20160520T053446_20160520T053513_011338_01134D_68E6'].First one will be used.
Multiple granules found for pair 20160508_20160520: ['S1A_IW_SLC__1SDV_20160520T053510_20160520T053537_011338_01134D_62C2', 'S1A_IW_SLC__1SDV_20160520T053446_20160520T053513_011338_01134D_68E6'].First one will be used.
Multiple granules found for pair 20160520_20160601: ['S1A_IW_SLC__1SDV_20160520T053510_20160520T053537_011338_01134D_62C2', 'S1A_IW_SLC__1SDV_20160520T053446_20160520T053513_011338_01134D_68E6'].First one will be used.
Multiple granules found for pair 20160520_20160707: ['S1A_IW_SLC__1SDV_20160520T053510_20160520T053537_011338_01134D_62C2', 'S1A_IW_SLC__1SDV_20160520T053446_20160520T053513_011338_01134D_68E6'].First one will be used.
Multiple granules found for pair 20160520_20160719: ['S1A_IW_SLC__1SDV_20160520T053510_20160520T053537_011338_01134D_62C2', 'S1A_IW_SLC__1SDV_20160520T053446_20160520T053513_011338_01134D_68E6'].First one will be used.
Multiple granules found for pair 20170509_20170527: ['S1A_IW_SLC__1SDV_20170527T053516_20170527T053543_016763_01BD9F_9756', 'S1A_IW_SLC__1SDV_20170527T053451_20170527T053518_016763_01BD9F_5D86'].First one will be used.
Multiple granules found for pair 20170515_20170527: ['S1A_IW_SLC__1SDV_20170527T053516_20170527T053543_016763_01BD9F_9756', 'S1A_IW_SLC__1SDV_20170527T053451_20170527T053518_016763_01BD9F_5D86'].First one will be used.
Multiple granules found for pair 20170521_20170527: ['S1A_IW_SLC__1SDV_20170527T053516_20170527T053543_016763_01BD9F_9756', 'S1A_IW_SLC__1SDV_20170527T053451_20170527T053518_016763_01BD9F_5D86'].First one will be used.
Multiple granules found for pair 20170527_20170602: ['S1A_IW_SLC__1SDV_20170527T053516_20170527T053543_016763_01BD9F_9756', 'S1A_IW_SLC__1SDV_20170527T053451_20170527T053518_016763_01BD9F_5D86'].First one will be used.
Multiple granules found for pair 20170527_20170608: ['S1A_IW_SLC__1SDV_20170527T053516_20170527T053543_016763_01BD9F_9756', 'S1A_IW_SLC__1SDV_20170527T053451_20170527T053518_016763_01BD9F_5D86'].First one will be used.
Multiple granules found for pair 20170527_20170614: ['S1A_IW_SLC__1SDV_20170527T053516_20170527T053543_016763_01BD9F_9756', 'S1A_IW_SLC__1SDV_20170527T053451_20170527T053518_016763_01BD9F_5D86'].First one will be used.
service1.flush_jobs()
service1
HyP3Service(
user_id=fanchengyan,
remaining_credits=655,
succeeded=0,
failed=0,
pending=553,
running=70
)
Download the succeeded jobs#
You can directly download the succeeded jobs using the InSARMission class. As historical jobs are also stored in the HyP3 database, you need to filter the jobs by the mission name or request_time. Following example demonstrates how to download the succeeded jobs with name="RGV_2024_03".
out_dir = "/Volumes/Data/GeoData/RGV/HyP3"
# download job missions from account 1
mission.download_jobs(out_dir, name="RGV_2024_03")
# download job missions from account 2
mission1.download_jobs(out_dir, name="RGV_2024_03")