Planing of repository layout

[soehms]

@kwankyu, I open this issue to find the structure of the new repository. From my point of view we need the following:

• ./LICENCE
  I would say GPL?
• ./README.md
  basic explanations, advantages and disadvantages of both approaches, links to the two .md files below. Maybe also links to directly download installation scripts from here
• ./sagemath_docker_guide.md
  my former README.md from projects_docker_guide
• ./sagemath_wsl_installer.md
  containing more information about your approach (file-names are just suggestions)
• ./src/
  containing .ps1 files from projects_docker_guide/src and in addition, if you like to extract a .ps1 template from your create-wsl-image.yml that could be here, too
• ./screenshots/
  shall we have sub-folders for both approaches?
• ./.github/workflows/
  containing the content of your create-wsl-image.yml and my docker_guide_installer_release.yml (which I created yesterday)
  maybe merged into one workflow (because they have the creation of the release note in common).

Anything else?

Previous work on this topic can be found at sagemath/sage-binder-env#20.

[kwankyu]

./icons/ containing icons for desktop shortcuts. Perhaps we should use sagemath official icons (if any) with the official sagemath logo. Or it may be ./src/icons/ if we put all parts that would make up the installers into one directory ./src/.

[kwankyu]

I don't know about licenses. ChatGPT recommends GPL v3 :-)

[kwankyu]

• ./screenshots/
  shall we have sub-folders for both approaches?

Yes. That would be clean.

[saraedum]

Are you planning to put my installer into here as well?

Source code is here: https://github.com/flatsurf/sage-flatsurf/tree/master/installer/win

plus this workflow file: https://github.com/flatsurf/sage-flatsurf/blob/master/.github/workflows/installer.yml

(I am happy to contribute this in a PR once the layout has been decided.)

[kwankyu]

• ./sagemath_docker_guide.md
  my former README.md from projects_docker_guide
• ./sagemath_wsl_installer.md
  containing more information about your approach (file-names are just suggestions)

You may name it sagemath_wsl_guide.md. But I am not sure how much content it would have. Putting all into the main README may suffice...

[soehms]

Are you planning to put my installer into here as well?

Sure! I think we should consolidate all inputs in this repo to achieve good solutions for Windows users.

Source code is here: https://github.com/flatsurf/sage-flatsurf/tree/master/installer/win

plus this workflow file: https://github.com/flatsurf/sage-flatsurf/blob/master/.github/workflows/installer.yml

That looks interesting!

(I am happy to contribute this in a PR once the layout has been decided.)

In general, I think we should work with PRs here. For a startup, in the first commit I created the layout as suggested here and added some files from my Docker Guide repo. Unfortunately, I can't do much more this week. Feel free to start without me.

[soehms]

Functionality we have in sum

FYI: I made some major changes in my approach. I got rid of Docker Desktop replacing it by a custom WSL distribution based on Alpine Linux. This makes the installation much easier, faster and less resource consuming and reduces startup-time.

How do we want to proceed here?

We have four codebases:

A) the one by @saraedum, which is developed for flatsurf and installs Sage as a custom WSL distribution.
B) the one by @kwankyu, which is developed in the sage-binder repository and installs Sage as a custom WSL distribution, too.
C) the one by @soehms, which is developed as an example for a projects docker guide and installs Sage via Docker in a custom WSL Docker distribution.
D) the one by @tobiasdiez, which is developed via a native Windows build of Sage using meson (see sagemath/sage#38872 and #1 (comment) below).

All except D) have in common that they need a working WSL installation and this is the hardest part of the installation and most sensitive to system requirements. Let me try to summarize what we have in total (please correct me if I'm wrong):

1. Installing WSL
    1.1 detection of state
        A) Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux /  VirtualMachinePlatform
        B) try .. catch  with wsl --install --no-distribution
        C) wsl --status
    1.2 installation
        A) Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux / VirtualMachinePlatform -NoRestart
        B) wsl --install --no-distribution
        C) wsl --install --no-distribution
    1.3 handling of reboot (if necessary)
        A) InnoSetup asks you to reboot and otherwise it doesn't continue. It adds the installer to the autostart so that the installation continues after the reboot.
        B) Print a hint that reboot could be necesarry.
        C) Query for reboot if it is necessary and perfom it if the user agrees. Give a hint that the installer must be restarted after reboot.
    1.4 detection that virtualization needs to be enabled in BIOS.
        A) -
        B) -
        C) Via result from wsl --import.
    1.5 Update WSL kernel
        A) Using msi wsl_update_x64.msi
        B) -
        C) -
2. Setting up the custom WSL distribution.
    A) Using wsldl to create a custom distribution for each Sage release from a tar-file created by Pixi (see link below)
    B) Using wsl --import to create a custom distribution for each Sage release from a tar-file exported from a Docker container.
    C) Using wsl --import to create a custom distribution with a running Docker daemon (Docker for Powershell) from a tar-file exported from a Docker container.
3. Installation method
    A) Install-Wizard created by Inno Setup
    B) Invoking a Powershell script
    C) Invoking a Powershell script
    D) not yet known
4. Launching Sage
    A) Create three desktop icons to launch an IPython, JupyterLab or Bash environment to work with Sage
    B) Create two desktop icons to launch an IPython or JupyterLab environment to work with Sage
    C) Create one desktop icon to launch a special application which allows to install arbitrary Sage releases from registries on Docker Hub and launch Ipython, Jupyter-Notebook (Lab needs an adaption in our Docker images) or Bash environments to work with Sage.
    D) not yet known

From a user perspective, there is hardly any difference between A) and B) (asuming equal resource consumption). Therefore, I think it makes sense to consolidate the two approaches. From a technical perspective, both approaches have different advantages:

A) Is implemented as part of installers for Linux, Mac and Windows and uses a common base via Pixi.
B) Is implemented in a more down-to-earth way and therefore easier to understand.

Is there a way to merge these two approaches? Maybe it would make sense to include 1.4 here as well. For C) it would be nice to have 1.5 and 3. (creating an install-wizard using Inno Setup) as per A). Is there a way to share the WSL installation across all three approaches?

[soehms]

Is there a way to merge these two approaches?

At least one thing could be done in common by following this suggestion by @tobiasdiez in sagemath/sage-binder-env#20 (comment) (regardless of whether we use Pixi or Docker containers to generate them):

I suggest to build the wsl tar image at https://github.com/sagemath/sage/blob/develop/.github/workflows/docker_hub.yml (or in a new workflow that is triggered by it). Not sure if there is a canonical place for the powershell script but it can just be in the root (or under docker). Or it goes in it's own repo as you have suggested.

[kwankyu]

I look forward to seeing one official windows installer that combines all the merits of the three approaches. The installer at sage-binder repo would be taken down when the installer here supersedes the former. I hope that this repo would be the canonical place visited by those looking for a windows installer for sage. The Sage documentation would direct people to here.

[tobiasdiez]

I'm also working on a native windows build of sage (using conda atm) in sagemath/sage#38872. It compiles but is definitely not yet production ready. Just wanted to mention it here as a "4." approach. Since it's not based on WSL, it circumvents many problems associated with this technique (but it comes with its own challenges of course). Maybe in the future it can be given as an experimental option in the "combined windows installer".

[soehms]

I'm also working on a native windows build of sage (using conda atm) in sagemath/sage#38872. It compiles but is definitely not yet production ready. Just wanted to mention it here as a "4." approach. Since it's not based on WSL, it circumvents many problems associated with this technique (but it comes with its own challenges of course). Maybe in the future it can be given as an experimental option in the "combined windows installer".

Great! It looks like a lot of work. I hope it will be successful and less fragile than the Cygwin approach. I added it as method D) above.

[saraedum]

Just to clarify 1.3.A) InnoSetup asks you to reboot and otherwise it doesn't continue. It adds the installer to the autostart so that the installation continues after the reboot. (All of this is a feature of InnoSetup.)

[saraedum]

Is there a way to share the WSL installation across all three approaches?

Windows installers are extremely weird. What InnoSetup can and cannot do at certain steps of the process and as which user things happen took a lot of effort to get (hopefully) right. Feel free to use the WSL prerequisites part in your installer of course, but I am not sure that it is going to be easy to put the rest of your process into InnoSetup as is. For example, I do not create the WSL distribution as part of the installer. I concluded that this is not really possible; instead you have to create the WSL distribution by launching the "installed program" as the current user once the actual installation is complete.

[soehms]

Just to clarify 1.3.A) InnoSetup asks you to reboot and otherwise it doesn't continue. It adds the installer to the autostart so that the installation continues after the reboot. (All of this is a feature of InnoSetup.)

Thanks! I corrected it in the list above!

[soehms]

Is there a way to share the WSL installation across all three approaches?

Windows installers are extremely weird. What InnoSetup can and cannot do at certain steps of the process and as which user things happen took a lot of effort to get (hopefully) right. Feel free to use the WSL prerequisites part in your installer of course, but I am not sure that it is going to be easy to put the rest of your process into InnoSetup as is. For example, I do not create the WSL distribution as part of the installer. I concluded that this is not really possible; instead you have to create the WSL distribution by launching the "installed program" as the current user once the actual installation is complete.

Thanks for these useful explanations and hints. I have to look in more detail into it, to see what is possible.

[soehms]

Current state after Sage 10.5 has been released

A) This seems to be broken. The installation abruptly stops without any messages (propably because of We already pressed Enter for you). Executing the Powershell script shows HRESULT: 0x80004005???:

PS C:\Program Files (x86)\sage-flatsurf-0.6.2> ./launch.ps1 --install
Installing virtual machine...
Installing Linux base system
Using: https://cloud-images.ubuntu.com/wsl/noble/current/ubuntu-noble-wsl-amd64-24.04lts.rootfs.tar.gz
Downloading...
 100% |███████████████████████████████████| (286/286 B)
Installing...
ERR: Unbekannter Fehler
HRESULT: 0x80004005
Press enter to exit...We already pressed Enter for you.
Setting up browser support
Es ist keine Verteilung mit dem angegebenen Namen vorhanden.
Fehlercode: Wsl/Service/WSL_E_DISTRO_NOT_FOUND
....

Obviously this is not due to the new Sage release. Moreover the installer now also fails for sage-flatsurf-0.5.2 where it worked before (see sagemath/sage#33765 (comment)). I had this on another machine, too.

B) Here the ps1-Files are missing for the 10.5 release. Using the installer for 10.4 after replacing the version number produces a working Sage 10.5.

C) still works because the 10.5 image was successfully built on Docker Hub.

D) is now in a state where the build should work (see sagemath/sage#38872 (comment) and sagemath/sage#38872 (comment)), but testing still needs to be done. I've started testing, but not yet successfully. I've probably made some mistakes. I'll report on it later.

I noticed a gap in the IPython app for B and C (I suspect also for A, but I haven't checked that yet). It's about the graphical output. This is fine for Jupyter Lab, but the IPython terminal cannot launch a graphical viewer because it was built on an Ubuntu server. One way to fix it would be to install graphical support in the container and use X11-forwarding. However, this would require almost 1 GB of additional space for the Docker images (tested with qimgv).

A less resource-intensive method would be to use a native Windows image viewer triggered via an appropriate socket. However, I don't have much experience with socket programming. Any hints for such an approach are very welcome.

[saraedum]

A) This cannot work because it needs SageMath 10.5 to be packaged on conda-forge. This has not happened yet.

I don't understand why it would fail for sage-flatsurf-0.5.2. This was actually fixed in flatsurf/sage-flatsurf@250bd87. The upstream problem in pixi has not been fixed yet. The next sage-flatsurf release will contain a workaround for this.

[tobiasdiez]

For D), you might experience sagemath/cysignals#212 - which is fixed in the most recent release (which you can already get from pypi). Once conda-forge/cysignals-feedstock#54 is merged, I'll update the conda lock files (@saraedum could you have a look at the open PRs in the cysignals feedstock please? Thanks!)

[soehms]

A) This cannot work because it needs SageMath 10.5 to be packaged on conda-forge. This has not happened yet.

What is the intended behavior if packaging of a new release is delayed on conda-forge? I asume it's not that the window is simply closed an nothing else happens.

I don't understand why it would fail for sage-flatsurf-0.5.2. This was actually fixed in flatsurf/sage-flatsurf@250bd87. The upstream problem in pixi has not been fixed yet. The next sage-flatsurf release will contain a workaround for this.

Thanks! I suspected an upstream issue. I'll keep an eye out for the next release.

[soehms]

For D), you might experience sagemath/cysignals#212 - which is fixed in the most recent release (which you can already get from pypi). Once conda-forge/cysignals-feedstock#54 is merged, I'll update the conda lock files (@saraedum could you have a look at the open PRs in the cysignals feedstock please? Thanks!)

Thanks for the hint, but I fear my problem starts earlier when setting up the correct build environment on Windows (I'm doing this for the first time after a long break). I will need some more time!

[soehms]

For D), you might experience sagemath/cysignals#212 - which is fixed in the most recent release (which you can already get from pypi)

@tobiasdiez, here my according attempt hoping it helps you to understand my problem:

(base) C:\Users\sebastian\sage>pip install cysignals
Collecting cysignals
  Downloading cysignals-1.12.2.tar.gz (65 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      + meson setup C:\Users\sebastian\AppData\Local\Temp\pip-install-zeq2ikd1\cysignals_b16781a105ac489f84500ac63237bf61 C:\Users\sebastian\AppData\Local\Temp\pip-install-zeq2ikd1\cysignals_b16781a105ac489f84500ac63237bf61\.mesonpy-g0lrys8i -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=C:\Users\sebastian\AppData\Local\Temp\pip-install-zeq2ikd1\cysignals_b16781a105ac489f84500ac63237bf61\.mesonpy-g0lrys8i\meson-python-native-file.ini
      The Meson build system
      Version: 1.6.1
      Source dir: C:\Users\sebastian\AppData\Local\Temp\pip-install-zeq2ikd1\cysignals_b16781a105ac489f84500ac63237bf61
      Build dir: C:\Users\sebastian\AppData\Local\Temp\pip-install-zeq2ikd1\cysignals_b16781a105ac489f84500ac63237bf61\.mesonpy-g0lrys8i
      Build type: native build
      Project name: cysignals
      Project version: undefined
      WARNING: Failed to activate VS environment: Could not parse vswhere.exe output

      ..\meson.build:1:0: ERROR: Unknown compiler(s): [['icl'], ['cl'], ['cc'], ['gcc'], ['clang'], ['clang-cl'], ['pgcc']]
      The following exception(s) were encountered:
      Running `icl ""` gave "[WinError 2] Das System kann die angegebene Datei nicht finden"
      Running `cl /?` gave "[WinError 2] Das System kann die angegebene Datei nicht finden"
      Running `cc --version` gave "[WinError 2] Das System kann die angegebene Datei nicht finden"
      Running `gcc --version` gave "[WinError 2] Das System kann die angegebene Datei nicht finden"
      Running `clang --version` gave "[WinError 2] Das System kann die angegebene Datei nicht finden"
      Running `clang-cl /?` gave "[WinError 2] Das System kann die angegebene Datei nicht finden"
      Running `pgcc --version` gave "[WinError 2] Das System kann die angegebene Datei nicht finden"

      A full log can be found at C:\Users\sebastian\AppData\Local\Temp\pip-install-zeq2ikd1\cysignals_b16781a105ac489f84500ac63237bf61\.mesonpy-g0lrys8i\meson-logs\meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

I got that in a Miniforge Prompt terminal, after I run succsessfully mamba env create --file environment-3.11-win.yml --name sage-dev there. It looks as if a connection between Miniforge and VS-Build-Tools is missing.

[tobiasdiez]

You need to run everything from the x64 Native Tools Command Prompt for VS 2022 and activate the conda env as the first step. There are some other minor issues (LIB not set correctly/automatically by conda), so best follow the instructions under the "Windows" tab in https://doc-pr-38872--sagemath.netlify.app/html/en/installation/meson.

Conda's compiler package actually tries to activate the vs build environment, but it's not working (see conda-forge/compilers-feedstock#66).

[soehms]

You need to run everything from the x64 Native Tools Command Prompt for VS 2022 and activate the conda env as the first step. There are some other minor issues (LIB not set correctly/automatically by conda), so best follow the instructions under the "Windows" tab in https://doc-pr-38872--sagemath.netlify.app/html/en/installation/meson.

Conda's compiler package actually tries to activate the vs build environment, but it's not working (see conda-forge/compilers-feedstock#66).

Thanks for the hints. See my testreport in sagemath/sage#38872 (comment)