iota2 feature maps

Assuming iota2 is fully operational, this chapter presents a usage of iota2: the production of features maps using satellite images time series.

Introduction to data

iota2 handles several sensors images :

  • Landsat 5 and 8 (old and new THEIA format)

  • Sentinel 1, Sentinel 2 L2A(THEIA and Sen2cor), Sentinel L3A (THEIA Format)

  • Various other images already processed, with the userFeat sensor

In this chapter, only the use of Sentinel 2 L2A will be illustrated. To use other sensors, it is necessary to adapt the input parameters.

To produce spectral indices, a user can use all spectral indices already computed by iota2 and provide python code to produce custom features.

Get the dataset

orphan:

nosearch:

Two datasets are available, containing minimal data required to run Iota2 builders:

  • A entire Sentinel 2 tile, with two dates (8.8 Go)

  • An extraction of Sentinel 2 data, with three dates over different eco-climatic region (Soon)

The archive contains:

  • /XXXX/IOTA2_TEST_S2
    archive content

    content of the tutorial archive after content extraction

    • ! external_code
      python code folder

      user custom python code used for external features / feature maps

      • external_code.py
        python code file

        contains user python code used to produce the spectral indices. Rules for creating user code are explained in external features page.

    • ! IOTA2_Outputs
      output folder

      folder used for iota2 output folders

      • (empty)

    • sensor_data
      input raster data

      the directory which contains Sentinel-2 data. These data must be stored by tiles as in the archive.

      • T31TCJ
        • ! SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7
          • MASKS

            • SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7_*.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7_FRE_B*.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7_FRE_STACK.tif

        • ! SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7
          • MASKS

            • SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7_*.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7_FRE_B*.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7_FRE_STACK.tif

      • T31TDJ
        • ! SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7
          • MASKS

            • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_*.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_ATB_R1.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_FRE_B*.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_FRE_STACK.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_MTD_ALL.xml

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_QKL_ALL.jpg

        • ! SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7
          • MASKS

            • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_*.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_ATB_R1.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_FRE_B*.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_FRE_STACK.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_MTD_ALL.xml

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_QKL_ALL.jpg

    • vector_data
      input vector data

      directory containing input vector data

      • reference_data.cpg

      • reference_data.dbf

      • reference_data.prj

      • reference_data.shp
        shapefile

        the shapeFile containing geo-referenced and labelled polygons (no multi-polygons, no overlapping) used to train a classifier.

      • reference_data.shx

      • EcoRegion.dbf

      • EcoRegion.prj

      • EcoRegion.qpj

      • EcoRegion.shp
        shapefile

        shapeFile containing two geo-referenced polygons representing a spatial stratification (eco-climatic areas, for instance).

      • EcoRegion.shx

    • colorFile.txt
      color table

      colors used in classification map

      $ cat colorFile.txt
...
211 255 85 0
...

      Here the class 211 has the RGB color 255 85 0.

    • IOTA2_Example.cfg
      example config file

      the file used to set iota2’s parameters such as inputs/outputs paths, classifier parameters etc.

    • i2_tutorial_classification.cfg
      sample config file

      sample config file for classification builder

    • i2_tutorial_features_map.cfg
      sample config file

      sample config file for feature map builder

    • i2_tutorial_obia.cfg
      sample config file

      sample config file for object base image analysis

    • nomenclature23.txt
      nomenclature file

      label’s name. The purpose of the file is to get a pretty results report at the end of the chain by relabeling integers labels by a more verbose type.

      & cat nomenclature.txt
...
prairie:211
...

      Here the class 211 corresponds to the class prairie

    • vecteur_23.qml

Understand the configuration file

iota2 exploits hundreds of parameters, native or coming from other libraries such as scikit-learn or OTB.

These combinations of parameters will make it possible to choose the processes to be carried out, the choice of the algorithms used or which output to write. A documentation of all these parameters is available: parameters descriptions.

To simplify the use, iota2 read a configuration file which is a simple text file containing sections and fields.

The minimal configuration file contains all required fields to produce a land cover map.

chain :
{
  output_path : '/XXXX/output_path_features'
  remove_output_path : True
  list_tile : 'T31TCJ'
  s2_path : '/XXXX/IOTA2_TEST_S2/sensor_data'
  first_step : 'init'
  last_step : 'mosaic'
  spatial_resolution : 10
  proj : 'EPSG:2154'
}

sensors_data_interpolation :
{
  auto_date : False
}
Sentinel_2 :
{
  start_date : '20180101'
  end_date : '20181231'
}
external_features :
{
  functions : 'get_cumulative_productivity get_minimum_productivity get_seasonal_variation'
  number_of_chunks : 25
  concat_mode : False
  no_data_value : -10000
}
builders :
{
  builders_class_name : ['I2FeaturesMap']
}
task_retry_limits :
{
  allowed_retry : 0
  maximum_ram : 180
  maximum_cpu : 40
}

For an end user, launching iota2 requires to fill correctly the configuration file.

In the above example, replace the XXXX by the path where the archive has been extracted.

Running the chain

iota2 launch

To launch the chain, simply use the following command line:

Iota2.py -config /XXXX/IOTA2_TESTS_DATA/i2_tutorial_features_map.cfg

First, the chain displays the list of all steps activated by the configuration file:

Group init:
         [x] Step 1: Sensors pre-processing
         [x] Step 2: Generate a common masks for each sensors
Group writing:
         [x] Step 3: Write features by chunk
         [x] Step 4: Merge features chunks per tile
Group mosaic:
         [x] Step 5: Merge features tiles into a mosaic

Once the processing start, a large amount of information will be printed, most of them concerning the dask-scheduler.

Did it all go well?

iota2 displays a lot of information during execution. It becomes difficult to identify if everything went well or if an error occurred.

To this end, in the output_path a log directory is available : output_path/logs.

The logs directory is ordered as:

├── CommonMasks
│        ├── common_mask_T31TCJ.err
│        └── common_mask_T31TCJ.out
├── html
│        └── *
├── MergeFeaturesMaps
│        ├── merge_feat_maps.err
│        └── merge_feat_maps.out
├── MergeFeaturesMapsByTiles
│        ├── merge_feat_maps_by_tiles_T31TCJ.err
│        └── merge_feat_maps_by_tiles_T31TCJ.out
├── run_informations.txt
├── SensorsPreprocess
│        ├── preprocessing_T31TCJ.err
│        └── preprocessing_T31TCJ.out
├── tasks_status_I2FeaturesMap_1.svg
├── tasks_status_I2FeaturesMap_2.svg
└── WriteFeaturesMap
    ├── write_features_T31TCJ_chunk_0.err
    ├── write_features_T31TCJ_chunk_0.out
    ├── write_features_T31TCJ_chunk_10.err
    ├── write_features_T31TCJ_chunk_10.out
    ├── write_features_T31TCJ_chunk_11.err
    ├── write_features_T31TCJ_chunk_11.out
         ├──         :    :    :
         ├──         :    :    :
    ├── write_features_T31TCJ_chunk_8.err
    ├── write_features_T31TCJ_chunk_8.out
    ├── write_features_T31TCJ_chunk_9.err
    └── write_features_T31TCJ_chunk_9.out

Each step has its own log directory. In these directories, two kinds of logs can be found: standard output and errors. The errors are compiled in “*.err” files and the standard output in “*.out” files. With the dask scheduler, iota2 goes as far as possible while the data required for the next steps is available. To simplify the error identification, an interactive graph is produced in a html page. Nodes in graph can have three colors (red: error, blue: done, white: not yielded). By clicking on graph node, the corresponding log file is opened.

If despite all this information, the errors can not be identified or solved, the iota2 team can help all users. The simplest way to ask help is to create an issue on framagit by adding the archive available in log directory.

Output tree structure

In this section, all output folders are described.

├── by_tiles
│   └── T31TCJ.tif
├── customF
│   ├── T31TCJ_chunk_0.tif
│   ├── T31TCJ_chunk_10.tif
│   ├── T31TCJ_chunk_11.tif
│   ├── T31TCJ_chunk_12.tif
│   ├── T31TCJ_chunk_13.tif
│   ├── T31TCJ_chunk_14.tif
│   ├── T31TCJ_chunk_15.tif
│   ├── T31TCJ_chunk_16.tif
│   ├── T31TCJ_chunk_17.tif
│   ├── T31TCJ_chunk_18.tif
│   ├── T31TCJ_chunk_19.tif
│   ├── T31TCJ_chunk_1.tif
│   ├── T31TCJ_chunk_2.tif
│   ├── T31TCJ_chunk_3.tif
│   ├── T31TCJ_chunk_4.tif
│   ├── T31TCJ_chunk_5.tif
│   ├── T31TCJ_chunk_6.tif
│   ├── T31TCJ_chunk_7.tif
│   ├── T31TCJ_chunk_8.tif
│   └── T31TCJ_chunk_9.tif
├── features
│   └── T31TCJ
│       └── tmp
│           ├── MaskCommunSL.dbf
│           ├── MaskCommunSL.prj
│           ├── MaskCommunSL.shp
│           ├── MaskCommunSL.shx
│           ├── MaskCommunSL.tif
│           ├── Sentinel2_T31TCJ_input_dates.txt
│           ├── Sentinel2_T31TCJ_interpolation_dates.txt
│           └── Sentinel2_T31TCJ_reference.tif
├── final
│   └── features_map.tif
├── IOTA2_tasks_status.txt
├── logs
│   ├── *
├── logs.zip
└── T31TCJ

customF

Temporary data used to reassemble maps for each tile. Can be removed by hand by user at the end of the processing.

by_tiles

The tile chunks are merged to produce a map for each tile.

final

This folder contains the final products of features map builder.

features

For each tile, contains useful information:

  • tmp/MaskCommunSL : the mask of all sensors for this tile

  • tmp/Sentinel2_T31TCJ_reference.tif : the image used for reprojecting data

  • tmp/Sentinel2_T31TCJ_input_dates.txt : the list of date detected in s2_path

  • tmp/Sentinel2_T31TCJ_interpolation_dates.txt : the list of output interpolation dates

Final products

Features map

features map

Features_map.tif Example

To go further