Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docs to support cloud storage for Cloud exports #6545

Merged
merged 8 commits into from
Jul 26, 2023
Merged

Docs to support cloud storage for Cloud exports #6545

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9465300
Docs to support cloud storage for Cloud exports
cwarnermm Jul 20, 2023
738feca
Updates based on feedback
cwarnermm Jul 21, 2023
b023b5c
Added new mmctl export command
cwarnermm Jul 21, 2023
bf7f10a
Broken link fix
cwarnermm Jul 21, 2023
60a37ca
refactoring of the cloud export part
JulienTant Jul 21, 2023
3e0b0d6
Merge branch 'master' into mm-53454
cwarnermm Jul 21, 2023
eced21e
Incorporated review feedback
cwarnermm Jul 21, 2023
aa8cc6f
Merge branch 'master' into mm-53454
cwarnermm Jul 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 55 additions & 30 deletions source/manage/cloud-data-export.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,23 @@ Migrate from Cloud to self-hosted
You can migrate your Cloud workspace data to a self-hosted deployment at any time.

How does the process work?
--------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~

Before you export and migrate your data, you must `install Mattermost </guides/deployment.html#install-guides>`_ on the server you’ll be using to run Mattermost. The migration is done using the mmctl CLI tool, which is a remote CLI tool for Mattermost that's installed locally and uses the Mattermost API. mmctl is pre-installed.

The `mmctl usage notes </manage/mmctl-command-line-tool.html#mmctl-usage-notes>`_ provide some additional context and information which you can reference before and during the process.

You'll be using the `mmctl export </manage/mmctl-command-line-tool.html#mmctl-export>`__ commands to export your Cloud data for channels, messages, users, etc. The export file is downloaded to a location specified in the export commands. Once the export is complete, you'll import the data into your self-hosted instance.
You'll be using the `mmctl export </manage/mmctl-command-line-tool.html#mmctl-export>`__ commands to export your Cloud data for channels, messages, users, etc. The export file is downloaded to a location specified in the export commands. Once the export is complete, you'll import the data into your self-hosted instance.

You can export the data to an Amazon S3 cloud storage location in cases where an export is quite large and challenging to download from the Mattermost server. See the `create the export <#create-the-export>`__ section below for details.

.. note::

Prior to migrating your data from Cloud, please ensure you have the appropriate permissions within your organization to carry out the data export which may contain sensitive information. Mattermost is not liable for any actions taken after the data export.

Moreover, the export process doesn’t include integrations or any custom data. Other aspects of your instance, such as specific security settings and requirements, are also not included. For assistance with migrating additional data and settings, see our support options: https://mattermost.com/support/.

Once mmctl is installed, we can generate the export from the source instance.
Once mmctl is authenticated, you can generate the export from the source instance.

Authenticate
------------
Expand All @@ -44,43 +46,66 @@ You'll be prompted for a username (use your admin user), password, and for a con
Create the export
-----------------

Once you're logged in, run:
Once you're logged in, run the mmctl export command based on your export storage location:

.. code::
.. tabs::

mmctl export create
.. tab:: Local export

This will create a full export of the server, and include attached files. Append ``--no-attachments`` if you do not wish to export attached files from your instance. This process can take some time, so ``mmctl`` will return immediately, and the job will run in the background on the Mattermost instance until the export is fully created. If successful, the command will immediately output a job ID, like this:
Run the following mmctl command:

.. code::
.. code::

mmctl export create

Export process job successfully created, ID: yfrr9ku5i7fjubeshs1ksrknzc
Running this command creates a full export of the server, including attached files. Append ``--no-attachments`` if you do not wish to export attached files from your instance. This process can take some time, so ``mmctl`` will return immediately, and the job will run in the background on the Mattermost instance until the export is fully created. If successful, the command will immediately output a job ID, like this:

While the job is running, its status can be checked using the ID that was provided when it was created, and when it's done the output will look similar to this:
.. code::

.. code::
Export process job successfully created, ID: yfrr9ku5i7fjubeshs1ksrknzc

mmctl export job show yfrr9ku5i7fjubeshs1ksrknzc
ID: yfrr9ku5i7fjubeshs1ksrknzc
Status: success
Created: 2021-11-03 10:44:13 -0500 CDT
Started: 2021-11-03 10:44:23 -0500 CDT
While the job is running, its status can be checked using the ID that was provided when it was created, and when it's done the output will look similar to this:

Download the export
-------------------
.. code::

Once the status is ``success``, you can download the export onto your local machine. First, discover the name of the completed export file with ``mmctl export list``:
mmctl export job show yfrr9ku5i7fjubeshs1ksrknzc
ID: yfrr9ku5i7fjubeshs1ksrknzc
Status: success
Created: 2021-11-03 10:44:13 -0500 CDT
Started: 2021-11-03 10:44:23 -0500 CDT

.. code::
Once the status is ``success``, download the export onto your local machine. First, discover the name of the completed export file with ``mmctl export list``:

mmctl export list
r3kcj8yuwbramdt714doafi3oo_export.zip
.. code::

This will show all of the exports on the server, so be sure to download the latest one and to delete it when you're done to save storage. Download the file with a command like the following, but with the filename of the export on your server:
mmctl export list
r3kcj8yuwbramdt714doafi3oo_export.zip

.. code::
This will show all of the exports on the server, so be sure to download the latest one and to delete it when you're done to save storage. Download the file with a command like the following, but with the filename of the export on your server:

.. code::

mmctl export download r3kcj8yuwbramdt714doafi3oo_export.zip

.. tab:: Cloud export

You can specify a dedicated cloud storage location for exports that's separate from file uploads. This option can be useful in cases where an export is quite large and challenging to download from the Mattermost server.

Run the following mmctl command:

.. code::

mmctl export generate-presigned-url [exportname]

Running this command creates a full export of the server, including attached files, in a unique Amazon S3 cloud storage location, and generates a pre-signed URL for the export file.

This process can take some time, so ``mmctl`` will return immediately, and the job will run in the background on the Mattermost instance until the export is fully created. If successful, the command will immediately output a job ID, like this:

.. code::

Export process job successfully created, ID: yfrr9ku5i7fjubeshs1ksrknzc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a section similar to that on the local export tab explaining how to show export job lists and how to check when the job is ready


mmctl export download r3kcj8yuwbramdt714doafi3oo_export.zip
From your Mattermost Cloud web instance, retrieve the file download link to the export by using the Mattermost slash command ``/exportlink [job-id|zip file|latest]``. Use the ``latest`` option to automatically pull the latest export available, or specify the download link by ``job-id`` or ``zip file``.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to be explicit as to where to run the slash command. Recommend we explain "hopping" over to the webapp to run the slash command and download the export.


Upload the export to the new server
-----------------------------------
Expand Down Expand Up @@ -166,7 +191,7 @@ If you’re using plugins that aren’t listed on the Marketplace, they won’t
The migration only includes data from Channels. No Boards or Playbooks data is exported.

Migration process
^^^^^^^^^^^^^^^^^
~~~~~~~~~~~~~~~~~

**Export from your self-hosted instance**

Expand Down Expand Up @@ -203,7 +228,7 @@ The completed file will be downloaded to your desktop as a ``.zip`` file.
The Support team will provide you with S3 credentials so you can upload the exported file. Once you’ve uploaded the file, please contact the Support team and let them know.

Create a new workspace on the Mattermost Cloud
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the meantime, you can log into Mattermost Cloud with your Mattermost credentials and create a Cloud workspace.

Expand All @@ -212,17 +237,17 @@ In the meantime, you can log into Mattermost Cloud with your Mattermost credenti
Do not create any users in your Mattermost Cloud instance as the migration process performs this task for you.

Importing your data into your Mattermost Cloud instance
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once the export upload to the provided S3 bucket is complete and you’ve shared your Mattermost Cloud instance name/URL, Support can begin the import step.

Depending on the size of the export this process can take some time. Support will contact you as soon as the import is complete. During this time it is highly recommended you do not use your Mattermost Cloud instance.

Start using your Mattermost Cloud instance
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When the export is complete, you can log into your Cloud instance and can invite your users to log in.

.. note::

It’s recommended that you keep your self-hosted Mattermost server until you’ve been using your Cloud instance for a while and all is verified as is as expected.
We recommend that you keep your self-hosted Mattermost server until you’ve been using your Cloud instance for a while and all is verified as is as expected.
36 changes: 35 additions & 1 deletion source/manage/mmctl-command-line-tool.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2248,6 +2248,7 @@ Manage exports.
- `mmctl export create`_ - Create an export file
- `mmctl export delete`_ - Delete an export file
- `mmctl export download`_ - Download export files
- `mmctl export generate-presigned-url`_ - Generate a pre-signed URL for an export file
- `mmctl export job`_ - List, show, and cancel export jobs
- `mmctl export job cancel`_ - Cancel export job
- `mmctl export job list`_ - List export jobs
Expand Down Expand Up @@ -2376,7 +2377,40 @@ Download export files.
--quiet prevent mmctl to generate output for the commands
--strict will only run commands if the mmctl version matches the server one
--suppress-warnings disables printing warning messages
mmctl export generate-presigned-url
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Description**
Generate a pre-signed URL for an export file in cases where a Mattermost Cloud export is large and challenging to download from the Mattermost server.
**Format**
.. code-block:: sh
mmctl export generate-presigned-url [exportname] [flags]
**Options**
.. code-block:: sh
-h, --help help for generate-presigned-url
**Options inherited from parent commands**
.. code-block:: sh
--config string path to the configuration file (default "$XDG_CONFIG_HOME/mmctl/config")
--disable-pager disables paged output
--insecure-sha1-intermediate allows to use insecure TLS protocols, such as SHA-1
--insecure-tls-version allows to use TLS versions 1.0 and 1.1
--json the output format will be in json format
--local allows communicating with the server through a unix socket
--quiet prevent mmctl to generate output for the commands
--strict will only run commands if the mmctl version matches the server one
--suppress-warnings disables printing warning messages
mmctl export job
~~~~~~~~~~~~~~~~
Expand All @@ -2388,7 +2422,7 @@ List, show, and export jobs.
.. code-block:: sh
-h, --help help for job
-h, --help help for job
**Options inherited from parent commands**
Expand Down