Use a single directory for all tests

[wigging]

Tests for the watertap package are located in 43 locations throughout the project as shown below.

watertap
    ├── core
    │   ├── plugins
    │   │   └── tests
    │   ├── tests
    │   └── util
    │       └── tests
    ├── costing
    │   └── tests
    ├── data
    │   └── techno_economic
    │       └── tests
    ├── edb
    │   └── tests
    ├── examples
    │   ├── chemistry
    │   │   └── tests
    │   ├── custom_model_demo
    │   │   └── tests
    │   ├── edb
    │   │   └── tests
    │   └── flowsheets
    │       ├── RO_with_energy_recovery
    │       │   └── tests
    │       ├── case_studies
    │       │   ├── activated_sludge
    │       │   │   └── tests
    │       │   ├── municipal_treatment
    │       │   │   └── tests
    │       │   ├── seawater_RO_desalination
    │       │   │   └── tests
    │       │   └── wastewater_resource_recovery
    │       │       ├── GLSD_anaerobic_digester
    │       │       │   └── tests
    │       │       ├── amo_1575_hrcs
    │       │       │   └── tests
    │       │       ├── amo_1575_magprex
    │       │       │   └── tests
    │       │       ├── amo_1595_photothermal_membrane_candoP
    │       │       │   └── tests
    │       │       ├── amo_1690
    │       │       │   └── tests
    │       │       ├── biomembrane_filtration
    │       │       │   └── tests
    │       │       ├── dye_desalination
    │       │       │   └── tests
    │       │       ├── electrochemical_nutrient_removal
    │       │       │   └── tests
    │       │       ├── groundwater_treatment
    │       │       │   └── tests
    │       │       ├── metab
    │       │       │   └── tests
    │       │       ├── peracetic_acid_disinfection
    │       │       │   └── tests
    │       │       ├── suboxic_activated_sludge_process
    │       │       │   └── tests
    │       │       ├── supercritical_sludge_to_gas
    │       │       │   └── tests
    │       │       └── swine_wwt
    │       │           └── tests
    │       ├── electrodialysis
    │       │   └── tests
    │       ├── full_treatment_train
    │       │   ├── analysis
    │       │   │   └── tests
    │       │   ├── flowsheet_components
    │       │   │   ├── chemistry
    │       │   │   │   └── tests
    │       │   │   └── tests
    │       │   └── model_components
    │       │       └── tests
    │       └── lsrro
    │           └── tests
    ├── property_models
    │   ├── activated_sludge
    │   │   └── tests
    │   └── tests
    ├── tests
    ├── tools
    │   ├── MPI
    │   │   └── tests
    │   └── parameter_sweep
    │       └── tests
    ├── ui
    │   └── tests
    └── unit_models
        ├── mvc
        │   ├── components
        │   │   └── tests
        │   └── tests
        ├── tests
        └── zero_order
            └── tests

To improve the maintainability of the WaterTAP project and the sanity of its developers, I suggest putting all the tests into a single top level directory.

watertap/
├── COPYRIGHT.md
├── LICENSE.md
├── README.md
├── bin/
├── codecov.yml
├── docs/
├── pytest.ini
├── requirements-dev.txt
├── requirements.txt
├── setup.py
├── tutorials/
├── tests/                    <<< All tests go here
└── watertap/

[ksbeattie]

This is an age-old discussion that we decided long ago to go with test being near code rather than grouped together.

[wigging]

So what is the benefit of having tests scattered throughout the code base? And why do you need to include the tests within the watertap package for each releases? Do you expect users to run the tests on their own system?

[lbianchi-lbl]

And why do you need to include the tests within the watertap package for each releases? Do you expect users to run the tests on their own system?

That's correct: by design, we want to have the ability of running as much of the WaterTAP's test suite within as many environments/platforms/installation types as possible. This allows us to ensure (to the extent of the specificity of our test suite) that both the WaterTAP code and the outputs of the models behave as expected within the very same environment in which the software will be used.

This is why we provide high-level steps for users to run tests: these are part of our Getting started guide, with (a reasonably close approximation of) the same steps being run as part of the PR CI check suite (https://github.com/watertap-org/watertap/blob/main/.github/workflows/checks.yml#L151-L202).

[wigging]

Ok, and can you elaborate on why it's better to put tests in multiple locations throughout the project?

[tarnold17]

I think the idea is that having tests near the code that they test makes it easier for developers and users to find any given test. That's not necessarily "better" than the other suggestion of having all tests in one location, just a different priority. I see pros and cons of doing it either way.

[wigging]

When a user installs a Python package, such as pip install numpy, that package and all of its source code gets installed in the site-packages directory. For example, in my conda environment that I use for watertap development, the numpy package is located at /opt/miniconda3/envs/watertap/lib/python3.8/site-packages/numpy/. Similarly, if a user does pip install watertap then everything in the watertap-org/watertap/watertap/ directory gets installed in that user's site-packages directory somewhere on their system. This includes all the tests which are located inside of the watertap package. Why do you want all these tests installed for users of the watertap package? Do you expect all the watertap users to also be developers of watertap?

[lbianchi-lbl]

The idea is that, in addition to its goal of detecting regressions when WaterTAP contributors make changes to the WaterTAP codebase, the test suite can also be used as a way to test that the end-to-end environment works correctly for the end user, on as many of the supported platforms as possible (or, if something doesn't work, knowing which test(s) are failing can let us pinpoint the underlying issue more effectively).

For this reason, having the tests be distributed alongside the main package is advantageous since it allows to run the tests directly whenever WaterTAP is installed by just running pytest --pyargs watertap without the users needing to download and configure the test suite as separate steps (see also #810 (reply in thread)) for more details).

[wigging]

But the users can't run the tests. They do pip install watertap. This does not install the dependencies needed to run the tests.

[lbianchi-lbl]

That's why we have the [testing] optional dependency, as documented in https://watertap.readthedocs.io/en/latest/getting_started.html#running-the-test-suite

[wigging]

Yes, that's right. But if someone installs without the [testing] or [dev] option then they don't have pytest. Therefore, they don't need those test files in the package because they're not running the tests. Is there a way to only install the test files when the package is installed with [testing] or [dev]?