Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated default readme. #221

Merged
merged 1 commit into from
Oct 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 20 additions & 85 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,122 +7,57 @@
![Github Pull Requests](https://img.shields.io/github/issues-pr/azure/osdu-developer) -->


OSDU Developer enables the deployment of personal instances of the [OSDU™](https://community.opengroup.org/osdu/platform) data platform.

For detailed instructions, view our online [Documentation](https://azure.github.io/osdu-developer/) and see what the team is currently working by looking through the [Roadmap](https://github.com/orgs/Azure/projects/696/views/2).

## OSDU Services

Supported services of OSDU are deployed from the current master branch. But can also deploy release branches of OSDU as specified in the [OSDU Milestones](https://community.opengroup.org/osdu/platform/-/milestones). (ie: release/0.25 release/0.26, release/0.27 etc.)

| **Core Services** | **Description** |
|-------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|
| [Partition Service](https://community.opengroup.org/osdu/platform/system/partition) | Manages data partitions to ensure efficient data management and scalability. |
| [Entitlement Service](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements) | Provides access control and permissions management for data within the OSDU platform. |
| [Legal Service](https://community.opengroup.org/osdu/platform/security-and-compliance/legal) | Ensures that data compliance and legal requirements are met, including data privacy and governance. |
| [Indexer Service](https://community.opengroup.org/osdu/platform/system/indexer-service) | Indexes and categorizes data to enable efficient search and retrieval. |
| [Indexer Queue](https://community.opengroup.org/osdu/platform/system/indexer-queue) | Manages the queue for processing indexing tasks, ensuring data is indexed in a timely manner. |
| [Schema Service](https://community.opengroup.org/osdu/platform/system/schema-service) | Manages and provides access to data schemas that define the structure and format of data. |
| [Storage Service](https://community.opengroup.org/osdu/platform/system/storage) | Provides scalable storage solutions for managing and retrieving large volumes of data. |
| [Search Service](https://community.opengroup.org/osdu/platform/system/search-service) | Facilitates searching and querying across data stored within the OSDU platform. |
| [File Service](https://community.opengroup.org/osdu/platform/system/file) | Handles file operations such as storage, retrieval, and management of data files. |
| [Workflow Service](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/) | Initiates business processes within the system. During the prototype phase, it facilitates CRUD operations on workflow metadata and triggers workflows in Apache Airflow. Additionally, the service manages process startup records, acting as a wrapper around Airflow functions.. |

| **Reference Helper Services** | **Description** |
|-------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|
| [Unit Service](https://community.opengroup.org/osdu/platform/system/reference/unit-service) | Provides dimension/measurement and unit definitions. |
| [CRS Catalog Service](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service) | Provides API endpoints to work with geodetic reference data, allowing developers to retrieve CRS definitions, select appropriate CRSs for data ingestion, and search for CRSs based on various constraints. |
| [CRS Conversion Service](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service) | Enables the conversion of coordinates from one coordinate reference system (CRS) to another. |

| **Airflow DAGS** | **Description** |
|-------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|
| [Manifest Ingestion DAG](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags) | Used for ingesting single or multiple metadata artifacts about datasets into OSDU. |
| [CSV Parser DAG:](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser) | Helps in parsing CSV files into a format for ingestion and processing. |

## Experimental Software

OSDU offers different experimental capabilities that are very new or community contributions. These services are not yet fully mature but are available for early adopters to test and provide feedback. This solution supports the concepts of experimental software with opt in feature flags.

| **Experimental Services** | **Description** |
|-------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|
| [Admin UI](https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui-totalenergies/admin-ui-totalenergies) | A community supported Angular Administration UI for OSDU. |


## Getting Started

> **IMPORTANT:** In order to deploy and run this example, you'll need an **Azure subscription** with [these namespaces](https://azure.github.io/osdu-developer/getting_started/#resource-providers/) registered.


### Prerequisites

#### To Run in GitHub Codespaces or VS Code Remote Containers

This solution can be run virtually by using GitHub Codespaces or VS Code Remote Containers _(Docker required)_. Click on one of the buttons below to open this repo in one of those options.

[![Open in Remote - Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Remote%20-%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/Azure/osdu-developer)
[![Open in GitHub Codespaces](https://img.shields.io/static/v1?style=for-the-badge&label=GitHub+Codespaces&message=Open&color=brightgreen&logo=github)](https://github.com/codespaces/new?skip_quickstart=true&machine=basicLinux32gb&repo=742135816&ref=main&devcontainer_path=.devcontainer%2Fdevcontainer.json&geo=UsEast)
[![Open in Remote - Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Remote%20-%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/Azure/osdu-developer)

#### To Run Locally

- __Visual Studio Code__: Install and configure with [REST Client Extension](https://marketplace.visualstudio.com/items?itemName=humao.rest-client) on your local machine. [Download](https://code.visualstudio.com/download)
This project provides simplified personal deployments of the [OSDU™](https://community.opengroup.org/osdu/platform) data platform on Microsoft Azure.

- __PowerShell Core__: Installed on your local machine. [Download](https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.4)

- __Azure CLI__: Installed on your local machine. [Download](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
- [Documentation](https://azure.github.io/osdu-developer/): Detailed concepts and architecture.
- [Services](https://azure.github.io/osdu-developer/services_source/): Current list of supported OSDU capabilities.
- [Tutorials](https://azure.github.io/osdu-developer/tutorial_cli/): Step-by-step guides for getting started
- [Roadmap](https://github.com/orgs/Azure/projects/696/views/2): Ongoing and future development plans

- __Azure Developer CLI__: Installed on your local machine. [Download](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/install-azd)

### CLI Deployment (Recommended)
### Getting Started

Deploying the resources via CLI is the recommended approach. This method allows for customization of the deployment parameters to better suit your needs, and offers tighter integration with other capabilities in this repository.
Review the [Getting Started](https://azure.github.io/osdu-developer/getting_started/) documentation prior to proceeding.

Follow this [tutorial](https://azure.github.io/osdu-developer/tutorial_cli/) for a quick overview of doing this from the Azure Cloud Shell.
### CLI Quickstart

> **NOTE:** Clone the latest version of this repo on your computer and switch to the directory. Note that the deployment scripts are being updated continously, make sure you update to the latest version.
> **Tutorial:** [Deploy OSDU Personal Instance via CLI](https://azure.github.io/osdu-developer/tutorial_cli/)

> **NOTE:** If you are using a pre-existing Entra ID App, you will need to set the feature flag `AZURE_CLIENT_ID` and `AZURE_CLIENT_SECRET` with the values from your app (See [Feature Flags](https://azure.github.io/osdu-developer/feature_flags/#custom-infrastructure) for more information on feature flags).
Clone the repository and run the following commands to deploy.

```bash
# Authentication
# Authenticate
az login --scope https://graph.microsoft.com//.default
az account set --subscription <your_subscription_id>
azd auth login

# Prepare Environment
azd init -e dev # This if first environment
azd env new dev # This if adding a new environment
azd env set <feature_flag> <value> # Set any necessary feature flags
# Prepare
azd init -e dev
azd env set <feature_flag> <value>

# Provisioning
azd provision

# Configure Settings for Integrations
# Configure
azd env set AUTH_CODE <auth_code>
azd hooks run settings

# Cleanup
azd down --force --purge
```

### Portal Quickstart _(reduced capabilities)_

### Portal Template Deployment (Alternative)
> **Tutorial:** [Deploy OSDU Personal Instance via Portal](https://azure.github.io/osdu-developer/tutorial_arm/)

[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fosdu-developer%2Fmain%2Fazuredeploy.json)

Deploying the solution is efficient and straightforward using an ARM (Azure Resource Manager) template. While this method utilizes default settings for ease of use, navigating parameter options and modifications can be challenging.

Follow this [tutorial](https://azure.github.io/osdu-developer/tutorial_arm/) for a quick overview of an ARM template deployment.
Deploy using the Azure Portal.

**Important Parameter Requirement:**

During the deployment process, some information is required to be provided in the deployment form:

- `Email Address`: A valid email address to be used as a first user.
- `Application Client Id`: Specify the Application Client Id. (The unique application ID of this application.)
- `Application Client Secret`: Specify the Application Client Secret. (A valid secret for the application client ID.)
- `Application Client Principal OID`: Specify the Enterprise Application Object Id. (The unique ID of the service principal object associated with the application.)
[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fosdu-developer%2Fmain%2Fazuredeploy.json)


## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Expand Down
1 change: 1 addition & 0 deletions docs/mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ nav:
- advanced_vnet.md
- Services:
- services_overview.md
- services_source.md
- OSDU:
- services_core.md
- services_reference.md
Expand Down
17 changes: 17 additions & 0 deletions docs/src/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,23 @@

Prerequisites and configuration steps for deploying personal OSDU™ instances in an Azure Subscription.


## Software Tools

!!! tip "Install Required Software"
Install the following software locally.

| Software | Description | Download Link |
|----------|-------------|---------------|
| Visual Studio Code | Code editor with REST Client Extension | [Download](https://code.visualstudio.com/download) |
| PowerShell Core | Cross-platform task automation solution | [Download](https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.4) |
| Azure CLI | Command-line tool for managing Azure resources | [Download](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) |
| Azure Developer CLI | Command-line tool for Azure development | [Download](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/install-azd) |

!!! note "Visual Studio Code Extension"
After installing Visual Studio Code, make sure to install the [REST Client Extension](https://marketplace.visualstudio.com/items?itemName=humao.rest-client).


## Subscription Quota

It is recommended to have at least 50 vCPUs in a region along with the ability to deploy Cosmos DB instances which can be resource constrained in some regions. Defaults for MSDN accounts can be increased by requesting a [quota increase](https://learn.microsoft.com/en-us/azure/quotas/regional-quota-requests).
Expand Down
Loading