Development recommendations ########################### GIT *** iota2 development uses `GIT `_ as source control tool. GIT-Flow ======== There are many ways of using GIT. In iota2 ``git-flow`` has been chosen to maintain the project (**without release branches**). You can find more about it `here `_. To roughly sum-up, for the development of a new feature, the developer **must** : 1. from ``develop``, create a feature ``branch`` 2. ``commit`` often 3. ``rebase`` (see below) new features coming from the ``develop`` branch into the feature branch 4. when the feature is ready, create a ``merge request`` into the develop branch. Commits must be prefixed with one of the following labels : +-------------+-------------------------------------------+ | Label | Purpose | +=============+===========================================+ | BUG FIX | Fix for runtime crash or incorrect result | +-------------+-------------------------------------------+ | INSTALL | Installation | +-------------+-------------------------------------------+ | DOC | Documentation update | +-------------+-------------------------------------------+ | ENH | Enhancement of an algorithm | +-------------+-------------------------------------------+ | NEW FEATURE | New functionality | +-------------+-------------------------------------------+ | PERF | Performance improvement | +-------------+-------------------------------------------+ | STYLE | No logic impact (indentation, comments) | +-------------+-------------------------------------------+ | WIP | Work In Progress not ready for merge | +-------------+-------------------------------------------+ Branch naming conventions ========================= +-----------------------+-------------------+------------------------------------------+ | branch purpose | naming convention | Example | +=======================+===================+==========================================+ | develop a new feature | feature-* | feature-Classifications_fusion | +-----------------------+-------------------+------------------------------------------+ | respond to an issue | issue-* | issue-#53_Improve_Classifications | +-----------------------+-------------------+------------------------------------------+ | fix a bug | fix-* | fix-#27_Bug | +-----------------------+-------------------+------------------------------------------+ The ``#27`` in *fix-#27_Bug* should refer to an issue reported on `iota2 GIT repository `_. Rebase your branch with develop =============================== To have a clear git tree a good practice is to use `rebase` instead of `merge`. This way all the work on the branch will appear after the last develop commit. It is important to understand that, for a simple rebase experience, you should avoid using standard merge into the feature branch. Once you have used `merge`, a rebase operation may fail. The standard procedure is therefore: 1. git checkout develop 2. git branch feature-branch 3. git checkout feature-branch 4. do your work and commit often 5. git rebase develop Here is a procedure to rebase a branch using emacs and `magit `_. Develop and custom branch are supposed to be clean and ready to merge (pulled, all changes commited...). Emacs is opened on the custom branch to rebase. .. code-block:: console # run magit M-x magit # enable rebase r # and choose elsewhere e # choose the branch to rebase with develop # then rebase start If there is no conflict the rebase is done automatically and your recent commits should look like: .. code-block:: console e30ee6eb custom_branch WIP: start to write rebase baseline ... 74ef6ea5 develop origin/develop Merge branch 'solve_test_error' into 'develop' If the custom branch is already pushed to the online repo, you must use `force-push` as you rewrite the history of the branch. In case of conflict ------------------- If a file is modified both in develop and in the custom branch, that probably leads to a conflict. In this case, the conflicting file(s) appear in the `Unstaged files` section in magit. By pressing `e` on a file, an ediff cession opens. Using `n` to move forward all conflicting lines, you can choose a variant by typing `a` or `b`. You can also manually solve the conflict by writing in the ediff buffer. Then quit ediff with `q` and save files as asked. .. warning:: Never commit a file until the end of rebase. Now you can restart the rebase operation with `r` and `r` to continue. Once ended, your commit history will be clean, and all develop commits will appear after the custom branch commits. .. note:: In order to avoid conflicts and to lighten the merge operations, it is important to make many small commits. It is then possible, once the rebase with develop is finished, to clean up the history of the branch by merging the commits and submit the merge request for integration in develop. FRAMAGIT ******** iota2 is hosted on `FramaGit `_ . Anybody can create an account for **free** and submit merge requests. To monitor the project, `issues `_ are mainly used. Developments are often started by a reported issue describing a bug, a new interesting feature or a research idea. Issues ====== If a developer wants to contribute to iota2, here is the recommended workflow: 1. If an issue about the new feature does not exist, create a dedicated one. 2. Assign the issue to herself. 3. When the contribution is done, close the issue. These simple rules will avoid duplicate work between developers. .. Note:: The board view is very useful to see which features are in development, need to be developed, or are in the backlog. .. figure:: ./Images/board_view.jpg :scale: 50 % :align: center :alt: Board view Board view CODE **** .. Warning:: Developers must use the pre-commit hook. It can be setup by the script enable_hooks.sh, which just makes a symlink into the ``.git/hooks`` directory. The pre-commit hook will format the code with ``yapf`` and run ``pylint`` and the pre-push hook will launch unit tests. .. _conda-install-dev: Installation ============ If you want to add modifications to iota2, it is highly recommended to install it using ``pixi`` instead of conda. It is easier to make modifications to the code, as well as adding dependencies. ``pixi`` also adds the possibility to implement quality of life (QoL) features for developers, such as ``pixi`` tasks. In the following paragraphs, we will assume you are using ``pixi``. However, if you still want to use conda, you will need to modify slightly your installation folder. First, ``cd`` to the conda environment folder. By default, it is located in your ``miniconda`` directory, which is installed (also by default) in your ``home`` directory. For example, if you called your environment ``iota2_dev``, it should be located here: ``~/miniconda3/envs/iota2_dev``. Supposing this path is right, go to ``~/miniconda3/envs/iota2_dev/lib/python3.12/site-packages/iota2``. Remove the iota2 directory (``rm -rf iota2``) and replace it with a symbolic link to where you cloned iota2: ``ln -s /path/to/iota2/source/code iota2``. Now, the modifications you add to the code will be taken into account when running ``Iota2.py``. Using pixi tasks ---------------- To run a ``pixi`` task, use ``pixi run task_name`` (for example ``pixi run tu``). You can also run specific tests with ``pixi run pytest iota2/tests/unittest/iota2_tests_sentinel2_usual_run.py`` for example. You can use different arguments to customize your pytest runs: * ``-n X`` uses ``X`` to run tests in parallel. * ``--reruns X --reruns-delay Y`` automatically reruns failed tests ``X`` times, waiting ``Y`` seconds between each rerun. * ``--cov iota2 --cov-report json:current_coverage.json`` produces a coverage report in json format. You can also show the report in the terminal or save it as HTML (see the ``pytest-cov`` `documentation `_). mypy ==== To improve the readability and maintainability of the code, we have chosen to use `mypy `_ with the following options : "--disallow-untyped-defs --follow-imports=skip --ignore-missing-imports". Before it can be integrated into iota2 via the merge request mechanism, the code must must comply as far as possible with mypy recommendations. If a recommendation cannot be applied, a discussion can be held on the code concerned. doc-string ========== Every useful function must contain a doc-string in NumPy style and using python `typing `_ in function signature. .. code-block:: python def func(arg1: int, arg2: str) -> bool: """Summary line. Extended description of function. Parameters ---------- arg1 : Description of arg1 arg2 : Description of arg2 Return ----- Description of return value: This function always return True, which is a boolean """ return True Citing code =========== Source code ----------- To reference source code in documentation you must use sphinx API link. The following rst code: .. code-block:: # First declare the current module with .. currentmodule:: iota2.common.custom_numpy_features Then create the link to a class with :class:`DataContainer` or a function :meth:`compute_custom_features` Produce the following documentation: .. currentmodule:: iota2.common.custom_numpy_features Then create the link to a class with :class:`DataContainer` or a function :meth:`compute_custom_features` Example code ------------ Creating and citing simple examples could be useful to explain algorithmic choices or function use. For this kind of citing, put the source file in `iota2/doc/source/examples`. Then you can cite the code: .. code-block:: Cite an entire class .. literalinclude:: examples/example.py :pyobject: example_class an entire function .. literalinclude:: examples/example.py :pyobject: example_method or text between two lines .. literalinclude:: examples/example.py :lines: 30-44 Produce the following documentation: Cite an entire class .. literalinclude:: examples/example.py :pyobject: ExampleClass an entire function .. literalinclude:: examples/example.py :pyobject: example_method or text between two lines .. literalinclude:: examples/example.py :lines: 30-44 TESTS ***** Pytest ====== iota2 is mainly developed in Python, and the `pytest `_ library has been chosen to implement our test framework. When adding modifications to the code, make sure you add tests relevant to these modifications. If you add a new feature, it must be tested. If you fix a bug that wasn't detected by previous tests, add a test to cover this case. Tests are separated in unit tests and integration tests. Unit tests are shorter and test specific functions. Integration tests simulate iota2 runs with different parameters. The test files are located in the ``/iota2/tests`` directory. To run a test or all tests from a directory, you can use ``pixi run pytest test_path`` or if you use conda ``pytest test_path``. If you use pixi, you can also use pre-defined tasks. At the moment, three exist: .. list-table:: Pixi tasks used for testing :widths: auto :header-rows: 1 * - Task name - Description - True command used * - ``tu``/``unit_tests`` - Runs only unit tests. - ``pytest $TEST_DIR/unittests -n 10`` * - ``ti``/``integration_tests`` - Runs only integration tests. - ``pytest $TEST_DIR/integrationtests -n 10`` * - ``full_tests_suite`` - | Runs the entire iota2 | test suite and measure | the coverage. - ``pytest $TEST_DIR --cov iota2 --cov-report json:current_coverage.json --cov-report html:current_coverage.html -n 10 --reruns 2 --reruns-delay 2`` .. Note:: To run a pixi task, use ``pixi run task_name`` (for example ``pixi run tu``). Baselines ========= A set of baseline data is already present in ``/iota2/data/references``. Please use them as references for your tests. If a new baseline must be created, add it in the directory previously quoted after discussion and consensus among the development team. .. Warning:: Baselines must be as small as possible. Code coverage ============= For the merge request to be accepted, the code coverage of the tests should be at least as good as the current code coverage of the branch ``develop``. For this, a code coverage analysis tool was created. First, run ``pytest --cov iota2 --cov-report json:code_coverage.json`` on your code. As we sue sub-processes, you need to follow these instructions: https://coverage.readthedocs.io/en/latest/subprocess.html. Otherwise, the measured coverage will be inaccurate and much lower than expected. The current coverage is provided here: ``iota/data/current_code_coverage.json``. After running tests with the coverage option on your code, you can use the provided tool (``iota2/common/tools/code_coverage_analysis.py``) to compare the difference of code coverage of the tests between what is already done on develop and the new branch. Two outputs are possible: a table printed in the terminal and an HTML file containing a table. This table contains the name of .py files, the current code coverage on develop, the code coverage of the new branch and the difference between them. For example: +----------+------------+------------------+-------------------------+ | File | Difference | Develop coverage | Incoming merge coverage | +==========+============+==================+=========================+ | file1.py | -5% | 80% | 75% | +----------+------------+------------------+-------------------------+ | file2.py | 6% | 70% | 76% | +----------+------------+------------------+-------------------------+ | file3.py | 40% | new file | 50% | +----------+------------+------------------+-------------------------+ | file4.py | 95% | new file | 95% | +----------+------------+------------------+-------------------------+ Four cases are possible: * The file already exists in develop: * The coverage is worse than what's already done in develop: file1 (not acceptable) * The coverage is the same or better: file is "acceptable": file2 (acceptable) * The file doesn't exists yet in develop: * The coverage doesn't meet the requirements (a threshold can be given before the analysis, 75% by default): file3 (not acceptable) * The coverage meets the requirements: file4 (acceptable) The user can also provide an "ignore threshold": some small changes in a file (adding/removing one line ...) may induce small changes in code coverage, which may represent a real downgrade of the coverage (0.2% by default). The inputs of the tool are: * ``develop_coverage``: Path to the develop's code coverage json file. * ``merge_coverage``: Path to the branch's code coverage json file. * ``new_files_threshold``: Threshold for new files' code coverage. * ``ignore_threshold``: Threshold for ignoring small changes in files' coverage. * ``html_file_path``: Path to the output HTML file (if not set, the file is not produced).