Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More AI docs updates #12515

Merged
merged 6 commits into from
Jan 20, 2025
Merged

More AI docs updates #12515

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6f9d909
fix: Update task_processing chatwithtools task type
marcelklehr Jan 17, 2025
50948b9
fix: Link to OpenAPI docs instead of documenting task_processing API …
marcelklehr Jan 17, 2025
1ec9b00
enh(admin/AI): Add docs for context_agent and text2image_stablediffus…
marcelklehr Jan 17, 2025
2549ee4
fix: Apply suggestions from code review
marcelklehr Jan 17, 2025
6eded4e
fix: Fix build error
marcelklehr Jan 17, 2025
c8bccba
fix: Fix AI toc tree
marcelklehr Jan 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions admin_manual/ai/ai_as_a_service.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ In order to use these providers you will need to install the respective app from

You can then add your API token and rate limits in the administration settings and set the providers live in the "Artificial intelligence" section of the admins settings.

Optionally but recommended, setup background workers for faster pickup of tasks. See :ref:`the relevant section in AI Overview<ai-overview_improve-ai-task-pickup-speed>` for more information.

OpenAI integration
------------------
Expand Down
13 changes: 12 additions & 1 deletion admin_manual/ai/app_assistant.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Text-To-Image

In order to make use of Text-To-Image features, you will need an app that provides an image generation backend:

* text2image_stablediffusion (Customer support available upon request)
* :ref:`tex2image_stablediffusion2<ai-app-text2image_stablediffusion2>` (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service<ai-ai_as_a_service>`)

Expand All @@ -97,6 +97,17 @@ In order to make use of our special Context Chat feature, offering in-context in

You will also need a text processing provider as specified above (ie. llm2 or integration_openai).

Context Agent
~~~~~~~~~~~~~

In order to make use of our AI agent feature, offering the execution of actions on behalf of the user based on the AI chat, you will need the following apps:


* :ref:`context_agent<ai-app-context_agent>` - (Customer support available upon request)


You will also need a text processing provider as specified above (ie. *llm2* or *integration_openai*).

Configuration
-------------

Expand Down
59 changes: 59 additions & 0 deletions admin_manual/ai/app_context_agent.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
==================================
App: Context Agent (context_agent)
==================================

.. _ai-app-context_agent:

The *context_agent* app is the apps that provide AI agent functionality in Nextcloud and acts as a backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>`. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
marcelklehr marked this conversation as resolved.
Show resolved Hide resolved

Requirements
------------

* This app is built as an External App and thus depends on AppAPI v3.1.0 or higher
* Nextcloud AIO is supported
* No GPU is necessary
marcelklehr marked this conversation as resolved.
Show resolved Hide resolved

* CPU Sizing

* At least 1GB of system RAM

Installation
------------

0. Make sure the :ref:`Nextcloud Assistant app<ai-app-assistant>` is installed
1. :ref:`Install AppAPI and setup a Deploy Demon<ai-app_api>`
2. Install the "Context Agent" ExApp via the "Apps" page in the Nextcloud web admin user interface
3. Install a text generation backend like :ref:`llm2 <ai-app-llm2>` or :ref:`integration_openai <ai-ai_as_a_service>` via the "Apps" page in Nextcloud


Model requirements
~~~~~~~~~~~~~~~~~~

This app requires underlying Large language models to support tool calling. The default model in *llm2* does *not* support tool calling. Instead we recommend:

* Qwen 2.5 8B or higher
* Watt Tool 8B or higher
Comment on lines +32 to +35
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can switch the default model in llm2 in favour of one of these models.
nothing again this PR though.


See :ref:`llm2 documentation <ai-app-llm2>` on how to configure alternate models.

Scaling
-------

It is currently not possible to scale this app, we are working on this.

App store
---------

You can also find the app in our app store, where you can write a review: `<https://apps.nextcloud.com/apps/context_agent>`_

Repository
----------

You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: `<https://github.com/nextcloud/context_agent>`_

Nextcloud customers should file bugs directly with our Support system.

Known Limitations
-----------------
* Make sure to test the language model you are using in concert with this app for whether they meet the use-case's quality requirements
* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI)
3 changes: 1 addition & 2 deletions admin_manual/ai/app_llm2.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,5 +129,4 @@ Known Limitations
* Language models are likely to generate false information and should thus only be used in situations that are not critical. It's recommended to only use AI at the beginning of a creation process and not at the end, so that outputs of AI serve as a draft for example and not as final product. Always check the output of language models before using it.
* Make sure to test the language model you are using it for whether it meets the use-case's quality requirements
* Language models notoriously have a high energy consumption, if you want to reduce load on your server you can choose smaller models or quantized models in exchange for lower accuracy
* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI)
* Due to technical limitations that we are in the process of mitigating, each task currently incurs a time cost of between 0 and 5 minutes in addition to the actual processing time
* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI)
60 changes: 60 additions & 0 deletions admin_manual/ai/app_text2image_stablediffusion2.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
=========================================================
App: Local Image Generation (text2image_stablediffusion2)
=========================================================

.. _ai-app-text2image_stablediffusion2:

The *text2image_stablediffusion2* app is one of the apps that provide image generation functionality in Nextcloud and act as an image generation backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>` and :ref:`other apps making use of the image generation functionality. The *text2image_stablediffusion2* app specifically runs only open source models and does so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.

Requirements
------------

* This app is built as an External App and thus depends on AppAPI v3.1.0 or higher
* Nextcloud AIO is supported
* We currently support NVIDIA GPUs and x86_64 CPUs
* CUDA >= v12.2 on your host system
* GPU Sizing

* A NVIDIA GPU with at least 8GB VRAM

* CPU Sizing

* At least 8GB of system RAM
* The more cores you have and the more powerful the CPU the better, we recommend 10-20 cores
* The app will hog all cores by default, so it is usually better to run it on a separate machine

Installation
------------

0. Make sure the :ref:`Nextcloud Assistant app<ai-app-assistant>` is installed
1. :ref:`Install AppAPI and setup a Deploy Demon<ai-app_api>`
2. Install the "Local large language model" ExApp via the "Apps" page in the Nextcloud web admin user interface
marcelklehr marked this conversation as resolved.
Show resolved Hide resolved


Scaling
-------

It is currently not possible to scale this app, we are working on this. Based on our calculations an instance has a rough capacity of 120 image requests per hour (each user request can be for multiple images). However, this number is based on theory and we do appreciate real-world feedback on this.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the testing setup could be mentioned here for a rough idea

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean some details of the machine on which 120 images/h was achieved, the gpu and cpu perhaps.


App store
---------

You can also find the app in our app store, where you can write a review: `<https://apps.nextcloud.com/apps/text2image_stablediffusion2>`_

Repository
----------

You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: `<https://github.com/nextcloud/text2image_stablediffusion2>`_

Nextcloud customers should file bugs directly with our Support system.

Known Limitations
-----------------

* The generated images are of a fixed resolution (512x512 pix), and the model does not achieve perfect photorealism
* The model cannot render legible text
* Faces and people in general may not be generated properly
* The results for certain image generation requests can be biased and may enforce stereotypes
* We currently only support languages that the underlying model supports; correctness of language use in languages other than English may be poor depending on the language's coverage in the model's training data
* Make sure to test the app for whether it meets the use-case's quality requirements
* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI)
180 changes: 1 addition & 179 deletions developer_manual/client_apis/OCS/ocs-taskprocessing-api.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,182 +8,4 @@ OCS TaskProcessing API

The OCS Task processing API allows you to run task processing tasks, like prompting large language models implemented by apps using :ref:`the backend Task Processing API<task_processing>`.

The base URL for all calls to this API is: ``<nextcloud_base_url>/ocs/v2.php/taskprocessing/``

All calls to OCS endpoints require the ``OCS-APIRequest`` header to be set to ``true``.


Get available task types
------------------------

.. versionadded:: 30.0.0

* Method: ``GET``
* Endpoint: ``/tasktypes``
* Response:
- Status code:
+ ``200 OK``
- Data:

+----------------------+--------+---------------------------------------------------------------------------------------------------------------+
| field | type | Description |
+----------------------+--------+---------------------------------------------------------------------------------------------------------------+
|``types`` | array | A map of supported task types. The keys are the task type IDs See below for the values. |
+----------------------+--------+---------------------------------------------------------------------------------------------------------------+

Task type fields:

+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+
| field | type | Description |
+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+
|``name`` | string | The name of the task type in the user's language |
+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+
|``description`` | string | A description of the task type in the user's language |
+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+
|``inputShape`` | array | The input shape of this task type |
+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+
|``outputShape`` | array | The output shape of this task type |
+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+

Input and output shape fields are maps from slot key to slot metadata:

+----------------------+--------+---------------------------------------------------------------------------------------------------------------+
| field | type | Description |
+----------------------+--------+---------------------------------------------------------------------------------------------------------------+
|``name`` | string | The name of the I/O slot |
+----------------------+--------+---------------------------------------------------------------------------------------------------------------+
|``description`` | string | A description of the I/O slot |
+----------------------+--------+---------------------------------------------------------------------------------------------------------------+
|``type`` | int | The I/O slot type (See backend API for the available types) |
+----------------------+--------+---------------------------------------------------------------------------------------------------------------+
|``mandatory`` | bool | Whether this slot is mandatory or not |
+----------------------+--------+---------------------------------------------------------------------------------------------------------------+

Schedule a task
---------------

.. versionadded:: 30.0.0

.. note:: The endpoint is rate limited as it can be quite resource intensive. Users can make 20 requests in 2 minutes, guests only 5

* Method: ``POST``
* Endpoint: ``/schedule``
* Data:

+-----------------+-------------+--------------------------------------------------------------------------------+
| field | type | Description |
+-----------------+-------------+--------------------------------------------------------------------------------+
|``input`` | array | The input text for the task |
+-----------------+-------------+--------------------------------------------------------------------------------+
|``type`` | string | Id of this task's type. |
+-----------------+-------------+--------------------------------------------------------------------------------+
|``appId`` | string | The id of the requesting app |
+-----------------+-------------+--------------------------------------------------------------------------------+
|``customId`` | string | An custom app-defined identifier for the task (optional) |
+-----------------+-------------+--------------------------------------------------------------------------------+

* Response:
- Status code:
+ ``200 OK``
+ ``400 Bad Request`` - When the task type is invalid
+ ``412 Precondition Failed`` - When the task type is not available currently
+ ``401 Unauthenticated`` - When the input references a file that the user doesn't have access to
+ ``429 Too Many Requests`` - When the rate limiting was exceeded

- Data:
+ ``input`` - Only provided in case of ``200 OK``, the task input, array
+ ``type`` - Only provided in case of ``200 OK``, the task type, string
+ ``id`` - Only provided in case of ``200 OK``, the assigned task id, int
+ ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API
+ ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string
+ ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string
+ ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string
+ ``output`` - Only provided in case of ``200 OK``, null
+ ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed

Fetch a task by ID
------------------

.. versionadded:: 30.0.0

.. note:: The endpoint is rate limited as it can be quite resource intensive. Users can make 20 requests in 2 minutes, guests only 5

* Method: ``POST``
* Endpoint: ``/task/{id}``

* Response:
- Status code:
+ ``200 OK``
+ ``404 Not Found`` - When the task could not be found

- Data:
+ ``input`` - Only provided in case of ``200 OK``, the task input, array
+ ``type`` - Only provided in case of ``200 OK``, the task type, string
+ ``id`` - Only provided in case of ``200 OK``, the assigned task id, int
+ ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API
+ ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string
+ ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string
+ ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string
+ ``output`` - Only provided in case of ``200 OK``, the output from the model, array or null
+ ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed


Cancel a task
-------------

.. versionadded:: 30.0.0

* Method: ``POST``
* Endpoint: ``/task/{id}/cancel``

* Response:
- Status code:
+ ``200 OK``
+ ``404 Not Found`` - When the task could not be found

- Data:
- Data:
+ ``input`` - Only provided in case of ``200 OK``, the task input, array
+ ``type`` - Only provided in case of ``200 OK``, the task type, string
+ ``id`` - Only provided in case of ``200 OK``, the assigned task id, int
+ ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API
+ ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string
+ ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string
+ ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string
+ ``output`` - Only provided in case of ``200 OK``, the output from the model, array or null
+ ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed


Delete a task
-------------

.. versionadded:: 30.0.0

* Method: ``DELETE``
* Endpoint: ``/task/{id}``

* Response:
- Status code:
+ ``200 OK``
+ ``404 Not Found`` - When the task could not be found

- Data:
+ ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed


Get task file contents
----------------------

.. versionadded:: 30.0.0

* Method: ``GET``
* Endpoint: ``/tasks/{id}/file/{fileId}``

* Response:
- Status code:
+ ``200 OK``
+ ``404 Not Found`` - When the task could not be found

- Data:
+ If ``200 OK`` this endpoint returns the raw data of the file
+ ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed
See the relevant `OpenAPI docs <https://docs.nextcloud.com/server/latest/developer_manual/_static/openapi.html#/operations/core-task_processing_api-schedule>`_.
Loading
Loading