.. _Vega supercomputer:

Vega supercomputer
==================

.. _Vega overview:

Overview
--------

`Vega <https://www.izum.si/en/vega-en/>`__
is a Tier-0, 10.1 PFLOPS peak performance,
Atos BullSequana supercomputer in Slovenia,
available to the University of Stuttgart
for the period 2023-2024 for development and benchmarking.

In total, the supercomputer consists of 1920 AMD EPYC Rome compute cores split
across 960 nodes and 240 NVIDIA Ampere A100 GPUs split across 60 nodes.
See the `Vega system overview <https://www.izum.si/en/vega-en/>`__
for more information.

+------------------+--------+-------+------------------+---------+---------------------+--------------+----------------------+
| Type             | Nodes  | Cores | RAM              | GPU     | Wall time           | Partition    | Nodelist             |
+==================+========+=======+==================+=========+=====================+==============+======================+
| CPU              | 960    | 2x64  | :qty:`256 <GB>`  |         | :qty:`2 <days>`     | ``cpu``      | ``cn[0001-0960]``    |
+------------------+--------+-------+------------------+---------+---------------------+--------------+----------------------+
| CPU large memory | 192    | 2x64  | :qty:`1 <TB>`    |         | :qty:`2 <days>`     | ``largemem`` | ``cn[0385-0576]``    |
+------------------+--------+-------+------------------+---------+---------------------+--------------+----------------------+
| CPU long jobs    | 6      | 2x64  | :qty:`256 <GB>`  |         | :qty:`4 <days>`     | ``cpu``      | ``cn[0010-0015]``    |
+------------------+--------+-------+------------------+---------+---------------------+--------------+----------------------+
| CPU dev jobs     | 8      | 2x32  | :qty:`256 <GB>`  |         | :qty:`30 <min>`     | ``dev``      | ``login[0001-0008]`` |
+------------------+--------+-------+------------------+---------+---------------------+--------------+----------------------+
| GPU              | 60     | 2x64  | :qty:`512 <GB>`  | 4x A100 | :qty:`2 <days>`     | ``gpu``      | ``gn[01-60]``        |
+------------------+--------+-------+------------------+---------+---------------------+--------------+----------------------+

This cluster is part of the EuroHPC JU infrastructure :cite:`axner24a`.

.. _Vega partitions:

Partitions and nodes
--------------------

.. _Vega CPU nodes:

CPU
^^^
There are 960 nodes equipped with 2 :cpu-amd-sheet2:`AMD EPYC Rome 7H12 <amd-epyc-7h12>`
(64 cores, 128 threads, :qty:`2.60 <GHz>`, :qty:`280 <W>`), :qty:`256 <GB>` of RAM DDR4-3200, :qty:`2 <TB>` M.2 SSD,
`Mellanox ConnectX-6 <https://nvdam.widen.net/s/5j7xtzqfxd/connectx-6-infiniband-datasheet-1987500-r2>`__
(:qty:`100 <Gbps>`) interconnect.
There are no GPUs.

CPU large memory
^^^^^^^^^^^^^^^^
There are 192 nodes among the :ref:`960 CPU nodes <Vega CPU nodes>` with :qty:`1 <TB>` of RAM DDR4-3200.

CPU long jobs
^^^^^^^^^^^^^
There are 6 nodes among the :ref:`960 CPU nodes <Vega CPU nodes>` with a 4-day time limit.

CPU dev jobs
^^^^^^^^^^^^
There are 8 nodes equipped with 2 :cpu-amd-sheet2:`AMD EPYC 7502 <amd-epyc-7502>`
(32 cores, 32 or 64 threads depending on the node, :qty:`2.50 <GHz>`, :qty:`180 <W>`), :qty:`256 <GB>` of RAM DDR4-3200.
There are no GPUs.

GPU
^^^
There are 60 nodes equipped with 2 :cpu-amd-sheet2:`AMD EPYC Rome 7H12 <amd-epyc-7h12>`
(64 cores, 128 threads, :qty:`2.60 <GHz>`, :qty:`280 <W>`), :qty:`512 <GB>` of RAM DDR4-3200, :qty:`2 <TB>` M.2 SSD,
`Mellanox ConnectX-6 <https://nvdam.widen.net/s/5j7xtzqfxd/connectx-6-infiniband-datasheet-1987500-r2>`__
(:qty:`100 <Gbps>`) interconnect.
Each node has 4 :gpu-nvidia-sheet1:`NVIDIA Ampere A100 PCIe
<Data-Center/a100/pdf/nvidia-a100-datasheet-nvidia-us-2188504-web.pdf>` (:qty:`64 <Gbps>`, :qty:`300 <W>`, :qty:`40 <GB>` HBM2e).

.. _Vega filesystems:

Filesystems
-----------

There are two types of mount points: :ref:`Lustre` for parallel computing
and :ref:`CephFS` for large file storage
(`link <https://en-vegadocs.vega.izum.si/mountpoints/>`__).

.. _Vega access:

Access
------

Use the EuroHPC portal (:ref:`instructions <EuroHPC access modes>`) to apply
for compute time. See `Proposal ID EHPC-DEV-2023D05-050
<https://gitlab.icp.uni-stuttgart.de/proposal/2023-hpc-vega-espresso-scalability>`__
for an example of a successful application for Development Access on Vega.

Once the proposal is approved, the Vega team will contact the project PI
and ask for a SSH public key and a GDPR form. Once access is granted,
refer to the `Vega user documentation <https://doc.vega.izum.si/>`__.
See also :doc:`/_pages/_appendix/using_vega` for building software and submitting jobs.

.. _Vega obligations:

Obligations
-----------

Usage of Vega comes with reporting obligations and acknowledgment obligations.
The exact rules are set by EuroHPC (:ref:`instructions <EuroHPC obligations>`).

.. _Vega publications:

Publications
------------

* :cite:`grad23a`: ICP simulations with 1024 cores on Vega
* :cite:`grad24a`: ICP simulations with 1024 cores on Vega