Skip to content

Commit

Permalink
update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
steven-mi committed Aug 14, 2023
1 parent 48b0252 commit 1707a02
Show file tree
Hide file tree
Showing 6 changed files with 59 additions and 31 deletions.
74 changes: 50 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,68 +1,94 @@
## DB-Rocket
## Databricks-Rocket

<img src="https://user-images.githubusercontent.com/2252355/173396060-8ebb3a33-f389-421d-bea4-afc01a078307.svg" width="100" height="100">

[![PyPI version](https://badge.fury.io/py/databricks-rocket.svg)](https://badge.fury.io/py/databricks-rocket)
![PyPI downloads](https://img.shields.io/pypi/dm/databricks-rocket)

Keep your local python scripts installed and in sync with a databricks notebook.
Every change on your local machine is automatically applied to the notebook.
Shortens the feedback loop to develop git based projects.
Removes the need to setup a local development environment.
Databricks-Rocket (short db-rockets), keeps your local Python scripts installed and synchronized with a Databricks notebook. Every change on your local machine
is automatically reflected in the notebook. This shortens the feedback loop for developing git-based projects and
eliminates the need to set up a local development environment.

## Installation

Install `databricks-rocket` using pip:

```sh
pip3 install databricks-rocket
pip install databricks-rocket
```

Please sure your python interpreter is 3.7 or higher.

## Setup

Make sure you have a token installed for databricks cli
([offical documentation](https://docs.databricks.com/dev-tools/cli/index.html)).
First, create a token for yourself on Databricks.
Then set it up locally by running:
Ensure you've created a personal access token in
Databricks ([offical documentation](https://docs.databricks.com/dev-tools/cli/index.html)). Afterward, set up the
Databricks CLI by executing:

```sh
databricks configure --token
```
Then, the databricks token must be exported in your environment.

Alternatively, you can set the Databricks token and host in your environment variables:

```sh
export DATABRICKS_HOST="mydatabrickshost"
export DATABRICKS_TOKEN="mydatabrickstoken"
```

If your project is not a pip package already you have to turn it into one. You can use dbrocket to do that.
If your project isn't already a pip package, you'll need to convert it into one. Use dbrocket for this:

```sh
rocket setup
```

Will create a setup.py for you.

## Using db-rocket
## Usage

### To Sync Your Project

By default, `databricks-rocket` syncs your project to DBFS automatically. This allows you to update your code and have
those changes reflected in your Databricks notebook without restarting the Python kernel. Simply execute:

```sh
rocket launch
```

The command returns the exact command you have to perform in your notebook next.

Example:
You'll then receive the exact command to run in your notebook. Example:

```sh
We are now building your Python repo as a library...
Done! in your notebook install the library by running:
stevenmi@MacBook db-rocket % rocket launch --watch=False
>> Watch activated. Uploaded your project to databricks. Install your project in your databricks notebook by running:
>> %pip install --upgrade pip
>> %pip install -r /dbfs/temp/stevenmi/db-rocket/requirements.txt
>> %pip install --no-deps -e /dbfs/temp/stevenmi/db-rocket

and following in a new Python cell:
>> %load_ext autoreload
>> %autoreload 2
```

%pip install --upgrade pip
%pip install /dbfs/temp/username/databricks_rocket-1.1.3-py3-none-any.whl --force-reinstall
Finally, add the content in you databricks notebook:
![imgs/img_2.png](imgs/img_2.png)

### To Upload Your Python Package

If you've disabled the watch feature, `databricks-rocket` will only upload your project as a wheel to DBFS:

```sh
rocket launch --watch=False
```

Create a cell in a notebook and paste the content (example below).
Example:

![img_1.png](img_1.png)
```sh
stevenmi@MacBook db-rocket % rocket launch --watch=False
>> Watch is disabled. Building creating a python wheel from your project
>> Found setup.py. Building python library
>> Uploaded ./dist/databricks_rocket-2.0.0-py3-none-any.whl to dbfs:/temp/stevenmi/db-rocket/dist/databricks_rocket-2.0.0-py3-none-any.whl
>> Uploaded wheel to databricks. Install your library in your databricks notebook by running:
>> %pip install --upgrade pip
>> %pip install /dbfs/temp/stevenmi/db-rocket/databricks_rocket-2.0.0-py3-none-any.whl --force-reinstall
```

## Support

Expand Down
File renamed without changes
File renamed without changes
Binary file added imgs/img_2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 4 additions & 5 deletions rocket/rocket.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@

import fire


from rocket.file_watcher import FileWatcher
from rocket.logger import logger
from rocket.utils import (
Expand Down Expand Up @@ -153,7 +152,8 @@ def _build_and_deploy(
and following in a new Python cell:
%load_ext autoreload
%autoreload 2""")
%autoreload 2"""
)
else:
logger.info(
f"""Watch activated. Uploaded your project to databricks. Install your project in your databricks notebook by running:
Expand Down Expand Up @@ -188,11 +188,10 @@ def _build_and_deploy(
if "index-url" in line
]
index_urls_options = " ".join(index_urls)

logger.info(
f"""Uploaded wheel to databricks. Install your library in your databricks notebook by running:
%pip install --upgrade pip
%pip install {index_urls_options} {install_path} --force-reinstall"""
%pip install --upgrade pip
%pip install {index_urls_options} {install_path} --force-reinstall"""
)

def _deploy(self, file_paths, dbfs_path, project_location):
Expand Down
7 changes: 5 additions & 2 deletions setup.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
import setuptools

# load the README file and use it as the long_description for PyPI
with open("README.md", encoding="utf8") as f:
readme = f.read()
try:
with open("README.md", encoding="utf8") as f:
readme = f.read()
except Exception as e:
readme = ""

setuptools.setup(
name="databricks-rocket",
Expand Down

0 comments on commit 1707a02

Please sign in to comment.