From f3477843922dabe43334d527ce2b1a10d18653be Mon Sep 17 00:00:00 2001 From: Synchon Mandal Date: Wed, 12 Jun 2024 18:28:04 +0200 Subject: [PATCH 1/2] docs: improve using/queueing.rst --- docs/using/queueing.rst | 87 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 80 insertions(+), 7 deletions(-) diff --git a/docs/using/queueing.rst b/docs/using/queueing.rst index 413ea09cb..174d02a90 100644 --- a/docs/using/queueing.rst +++ b/docs/using/queueing.rst @@ -10,10 +10,10 @@ computational clusters. This is done by adding the ``queue`` section in the :ref:`codeless` file and executing the ``junifer queue`` command. While junifer is meant to support `HTCondor`_, `SLURM`_ and local queueing -using `GNU Parallel`_, only HTCondor is currently supported. This will be -implemented in future releases of ``junifer``. If you are in immediate need of -any of these schedulers, please create an issue on the `junifer github`_ -repository. +using `GNU Parallel`_, only ``HTCondor`` and ``GNU Parallel`` are currently +supported. This will be implemented in future releases of ``junifer``. If you are +in immediate need of any of these schedulers, please create an issue on the +`junifer github`_ repository. The ``queue`` section of the :ref:`codeless` must start by defining the following general parameters: @@ -22,8 +22,8 @@ following general parameters: folder where the job files will be created, as well as any relevant file. Depending on the scheduler, it will also be listed in the queueing system with this name. -* ``kind``: The kind of scheduler to be used. Currently, only ``HTCondor`` is - supported. +* ``kind``: The kind of scheduler to be used. Currently, only ``HTCondor`` and + ``GNUParallelLocal`` are supported. Example in YAML: @@ -48,7 +48,9 @@ jobs are finished. The following parameters are available for HTCondor: -* ``env``: Definition of the Python environment. It must provide two variables: +* ``pre_run``: Extra shell commands to run before ``junifer run``. +* ``pre_collect``: Extra shell commands to run before ``junifer collect``. +* ``env``: Definition of the Python environment. It has the following parameters: * ``kind``: This is the kind of virtual environment to use: @@ -61,6 +63,9 @@ The following parameters are available for HTCondor: the absolute or relative path to the virtualenv when ``venv`` is used. If relative path is used then it should be relative to the YAML. + * ``shell``: This is the shell to use. Only ``bash`` and ``zsh`` are supported + as of now. + * ``mem``: Memory to be used by the job. It must be provided as a string with the units (e.g., ``"2GB"``). * ``cpus``: Number of CPUs to be used by the job. It must be provided as an @@ -93,10 +98,78 @@ Example in YAML: env: kind: conda name: junifer + shell: zsh mem: 8G disk: 2G collect: "yes" # wrap it in string to avoid boolean +Alternatively, if you use ``venv``, the YAML would look like so: + +.. code-block:: yaml + + queue: + jobname: TestHTCondorQueue + kind: HTCondor + env: + kind: venv + name: /home/me/junifer-venv # should be at the level above `bin/activate` + shell: zsh + mem: 8G + disk: 2G + collect: "yes" # wrap it in string to avoid boolean + +.. _queueing_local: + +GNUParallelLocal +---------------- + +When using GNU Parallel, ``junifer`` will run only in local mode and leverage +maximum computational power of whatever hardware it is run on. + +The following parameters are available for GNUParallelLocal: + +* ``pre_run``: Extra shell commands to run before ``junifer run``. +* ``pre_collect``: Extra shell commands to run before ``junifer collect``. +* ``env``: Definition of the Python environment. It has the following parameters: + + * ``kind``: This is the kind of virtual environment to use: + + * ``conda`` + * ``venv`` + * ``local`` (no virtual environment) + + * ``name``: This is the name of the environment to use in case a virtual + environment is used. It should be the name when ``conda`` is used and + the absolute or relative path to the virtualenv when ``venv`` is used. + If relative path is used then it should be relative to the YAML. + + * ``shell``: This is the shell to use. Only ``bash`` and ``zsh`` are supported + as of now. + +Example in YAML: + +.. code-block:: yaml + + queue: + jobname: TestGNUParallelLocalQueue + kind: GNUParallelLocal + env: + kind: conda + name: junifer + shell: zsh + +Alternatively, if you use ``venv``, the YAML would look like so: + +.. code-block:: yaml + + queue: + jobname: TestGNUParallelLocalQueue + kind: GNUParallelLocal + env: + kind: venv + name: /home/me/junifer-venv # should be at the level above `bin/activate` + shell: zsh + Once the :ref:`codeless` file is ready, including the ``queue`` section, you can queue the jobs by executing the ``junifer queue`` command. -- 2.52.0 From 545c4163f88067e81cf16f5f67171e9495d8541b Mon Sep 17 00:00:00 2001 From: Synchon Mandal Date: Wed, 19 Jun 2024 10:28:10 +0200 Subject: [PATCH 2/2] chore: add changelog 327.doc --- docs/changes/newsfragments/327.doc | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/changes/newsfragments/327.doc diff --git a/docs/changes/newsfragments/327.doc b/docs/changes/newsfragments/327.doc new file mode 100644 index 000000000..04a2729e8 --- /dev/null +++ b/docs/changes/newsfragments/327.doc @@ -0,0 +1 @@ +Update "Queueing Jobs (HPC, HTC)" section by `Synchon Mandal`_ -- 2.52.0