Development recommendations

GIT

iota2 development uses GIT as source control tool.

GIT-Flow

There are many ways of using GIT. In iota2 git-flow has been chosen to maintain the project (without release branches). You can find more about it here.

To roughly sum-up, for the development of a new feature, the developer must :

from develop, create a feature branch

commit often

rebase (see below) new features coming from the develop branch into the feature branch

when the feature is ready, create a merge request into the develop branch.

Commits must be prefixed with one of the following labels :

Label	Purpose
BUG FIX	Fix for runtime crash or incorrect result
INSTALL	Installation
DOC	Documentation update
ENH	Enhancement of an algorithm
NEW FEATURE	New functionality
PERF	Performance improvement
STYLE	No logic impact (indentation, comments)
WIP	Work In Progress not ready for merge

Branch naming conventions

branch purpose	naming convention	Example
develop a new feature	feature-*	feature-Classifications_fusion
respond to an issue	issue-*	issue-#53_Improve_Classifications
fix a bug	fix-*	fix-#27_Bug

The #27 in fix-#27_Bug should refer to an issue reported on iota2 GIT repository.

Rebase your branch with develop

To have a clear git tree a good practice is to use rebase instead of merge. This way all the work on the branch will appear after the last develop commit.

It is important to understand that, for a simple rebase experience, you should avoid using standard merge into the feature branch. Once you have used merge, a rebase operation may fail.

The standard procedure is therefore:

git checkout develop
git branch feature-branch
git checkout feature-branch
do your work and commit often
git rebase develop

Here is a procedure to rebase a branch using emacs and magit. Develop and custom branch are supposed to be clean and ready to merge (pulled, all changes commited…). Emacs is opened on the custom branch to rebase.

# run magit
M-x magit
# enable rebase
r
# and choose elsewhere
e
# choose the branch to rebase with
develop
# then rebase start

If there is no conflict the rebase is done automatically and your recent commits should look like:

e30ee6eb custom_branch WIP: start to write rebase baseline
...
74ef6ea5 develop origin/develop Merge branch 'solve_test_error' into 'develop'

If the custom branch is already pushed to the online repo, you must use force-push as you rewrite the history of the branch.

In case of conflict

If a file is modified both in develop and in the custom branch, that probably leads to a conflict. In this case, the conflicting file(s) appear in the Unstaged files section in magit. By pressing e on a file, an ediff cession opens. Using n to move forward all conflicting lines, you can choose a variant by typing a or b. You can also manually solve the conflict by writing in the ediff buffer. Then quit ediff with q and save files as asked.

Warning

Never commit a file until the end of rebase.

Now you can restart the rebase operation with r and r to continue.

Once ended, your commit history will be clean, and all develop commits will appear after the custom branch commits.

Note

In order to avoid conflicts and to lighten the merge operations, it is important to make many: small commits. It is then possible, once the rebase with develop is finished, to clean up the history of the branch by merging the commits and submit the merge request for integration in develop.

FRAMAGIT

iota2 is hosted on FramaGit . Anybody can create an account for free and submit merge requests. To monitor the project, issues are mainly used. Developments are often started by a reported issue describing a bug, a new interesting feature or a research idea.

Issues

If a developer wants to contribute to iota2, here is the recommended workflow:

If an issue about the new feature does not exist, create a dedicated one.

Assign the issue to herself.

When the contribution is done, close the issue.

These simple rules will avoid duplicate work between developers.

Note

The board view is very useful to see which features are in development, need to be developed, or are in the backlog.

CODE

Warning

Developers must use the pre-commit hook. It can be setup by the script enable_hooks.sh, which just makes a symlink into the .git/hooks directory. The pre-commit hook will format the code with yapf and run pylint and the pre-push hook will launch unit tests.

Installation

If you want to add modifications to iota2, it is highly recommended to install it using pixi instead of conda. It is easier to make modifications to the code, as well as adding dependencies. pixi also adds the possibility to implement quality of life (QoL) features for developers, such as pixi tasks. In the following paragraphs, we will assume you are using pixi.

However, if you still want to use conda, you will need to modify slightly your installation folder. First, cd to the conda environment folder. By default, it is located in your miniconda directory, which is installed (also by default) in your home directory. For example, if you called your environment iota2_dev, it should be located here: ~/miniconda3/envs/iota2_dev. Supposing this path is right, go to ~/miniconda3/envs/iota2_dev/lib/python3.12/site-packages/iota2. Remove the iota2 directory (rm -rf iota2) and replace it with a symbolic link to where you cloned iota2: ln -s /path/to/iota2/source/code iota2. Now, the modifications you add to the code will be taken into account when running Iota2.py.

Using pixi tasks

To run a pixi task, use pixi run task_name (for example pixi run tu). You can also run specific tests with pixi run pytest iota2/tests/unittest/iota2_tests_sentinel2_usual_run.py for example. You can use different arguments to customize your pytest runs:

-n X uses X to run tests in parallel.
--reruns X --reruns-delay Y automatically reruns failed tests X times, waiting Y seconds between each rerun.
--cov iota2 --cov-report json:current_coverage.json produces a coverage report in json format. You can also show the report in the terminal or save it as HTML (see the pytest-cov documentation).

mypy

To improve the readability and maintainability of the code, we have chosen to use mypy with the following options : “–disallow-untyped-defs –follow-imports=skip –ignore-missing-imports”. Before it can be integrated into iota2 via the merge request mechanism, the code must must comply as far as possible with mypy recommendations. If a recommendation cannot be applied, a discussion can be held on the code concerned.

doc-string

Every useful function must contain a doc-string in NumPy style and using python typing in function signature.

def func(arg1: int, arg2: str) -> bool:
    """Summary line.

    Extended description of function.

    Parameters
    ----------
    arg1 :
        Description of arg1
    arg2 :
        Description of arg2

    Return
    -----
        Description of return value:
        This function always return True, which is a boolean

    """
    return True

Citing code

Source code

To reference source code in documentation you must use sphinx API link.

The following rst code:

# First declare the current module with
.. currentmodule:: iota2.common.custom_numpy_features

Then create the link to a class with :class:`DataContainer`
or a function :meth:`compute_custom_features`

Produce the following documentation:

Then create the link to a class with DataContainer or a function compute_custom_features()

Example code

Creating and citing simple examples could be useful to explain algorithmic choices or function use. For this kind of citing, put the source file in iota2/doc/source/examples.

Then you can cite the code:

Cite an entire class

.. literalinclude:: examples/example.py
   :pyobject: example_class

an entire function

.. literalinclude:: examples/example.py
   :pyobject: example_method

or text between two lines

.. literalinclude:: examples/example.py
   :lines: 30-44

Produce the following documentation:

Cite an entire class

class ExampleClass:
    """Empty class"""

    def __init__(self):
        """Do nothing"""

    def meth1(self):
        """Example method"""

an entire function

def example_method(arg1: int, arg2: int) -> int:
    """
    This method sum the two args

    Parameters
    ----------

    arg1:
        a integer value
    arg2:
        a integer value

    Notes
    -----
        The output value is the sum of the two inputs
    """
    return arg1 + arg2

or text between two lines

    This method sum the two args

    Parameters
    ----------

    arg1:
        a integer value
    arg2:
        a integer value

    Notes
    -----
        The output value is the sum of the two inputs
    """
    return arg1 + arg2

TESTS

Pytest

iota2 is mainly developed in Python, and the pytest library has been chosen to implement our test framework. When adding modifications to the code, make sure you add tests relevant to these modifications. If you add a new feature, it must be tested. If you fix a bug that wasn’t detected by previous tests, add a test to cover this case.

Tests are separated in unit tests and integration tests. Unit tests are shorter and test specific functions. Integration tests simulate iota2 runs with different parameters. The test files are located in the /iota2/tests directory. To run a test or all tests from a directory, you can use pixi run pytest test_path or if you use conda pytest test_path. If you use pixi, you can also use pre-defined tasks. At the moment, three exist:

Pixi tasks used for testing
Task name	Description	True command used
`tu`/`unit_tests`	Runs only unit tests.	`pytest $TEST_DIR/unittests -n 10`
`ti`/`integration_tests`	Runs only integration tests.	`pytest $TEST_DIR/integrationtests -n 10`
`full_tests_suite`	Runs the entire iota2 test suite and measure the coverage.	`pytest $TEST_DIR --cov iota2 --cov-report json:current_coverage.json --cov-report html:current_coverage.html -n 10 --reruns 2 --reruns-delay 2`

Note

To run a pixi task, use pixi run task_name (for example pixi run tu).

Baselines

A set of baseline data is already present in /iota2/data/references. Please use them as references for your tests. If a new baseline must be created, add it in the directory previously quoted after discussion and consensus among the development team.

Warning

Baselines must be as small as possible.

Code coverage

For the merge request to be accepted, the code coverage of the tests should be at least as good as the current code coverage of the branch develop. For this, a code coverage analysis tool was created. First, run pytest --cov iota2 --cov-report json:code_coverage.json on your code. As we sue sub-processes, you need to follow these instructions: https://coverage.readthedocs.io/en/latest/subprocess.html. Otherwise, the measured coverage will be inaccurate and much lower than expected. The current coverage is provided here: iota/data/current_code_coverage.json.

After running tests with the coverage option on your code, you can use the provided tool (iota2/common/tools/code_coverage_analysis.py) to compare the difference of code coverage of the tests between what is already done on develop and the new branch. Two outputs are possible: a table printed in the terminal and an HTML file containing a table. This table contains the name of .py files, the current code coverage on develop, the code coverage of the new branch and the difference between them. For example:

File	Difference	Develop coverage	Incoming merge coverage
file1.py	-5%	80%	75%
file2.py	6%	70%	76%
file3.py	40%	new file	50%
file4.py	95%	new file	95%

Four cases are possible:

The file already exists in develop:
- The coverage is worse than what’s already done in develop: file1 (not acceptable)
- The coverage is the same or better: file is “acceptable”: file2 (acceptable)
The file doesn’t exists yet in develop:
- The coverage doesn’t meet the requirements (a threshold can be given before the analysis, 75% by default): file3 (not acceptable)
- The coverage meets the requirements: file4 (acceptable)

The user can also provide an “ignore threshold”: some small changes in a file (adding/removing one line …) may induce small changes in code coverage, which may represent a real downgrade of the coverage (0.2% by default).

The inputs of the tool are:

develop_coverage: Path to the develop’s code coverage json file.
merge_coverage: Path to the branch’s code coverage json file.
new_files_threshold: Threshold for new files’ code coverage.
ignore_threshold: Threshold for ignoring small changes in files’ coverage.
html_file_path: Path to the output HTML file (if not set, the file is not produced).