package management (dbt-labs#542)

* semver resolution * cleanup * remove unnecessary comment * add test for multiples on both sides * add resolve_to_specific_version * local registry * hacking out deps * Buck pkg mgmt (dbt-labs#645) * only load hooks and archives once (dbt-labs#540) * sets schema for node before parsing raw sql (dbt-labs#541) * Fix/env vars (dbt-labs#543) * fix for bad env_var exception * overwrite target with compiled values * fixes env vars, adds test. Auto-compile profile/target args * improvements for code that runs in hooks (dbt-labs#544) * improvements for code that runs in hooks * fix error message note * typo * Update CHANGELOG.md * bump version (dbt-labs#546) * add scope to service account json creds initializer (dbt-labs#547) * bump 0.9.0a3 --> 0.9.0a4 (dbt-labs#548) * Fix README links (dbt-labs#554) * Update README.md * handle empty profiles.yml file (dbt-labs#555) * return empty string (instead of None) to avoid polluting rendered sql (dbt-labs#566) * tojson was added in jinja 2.9 (dbt-labs#563) * tojson was added in jinja 2.9 * requirements * fix package-defined schema test macros (dbt-labs#562) * fix package-defined schema test macros * create a dummy Relation in parsing * fix for bq quoting (dbt-labs#565) * bump snowflake, remove pyasn1 (dbt-labs#570) * bump snowflake, remove pyasn1 * change requirements.txt * allow macros to return non-text values (dbt-labs#571) * revert jinja version, implement tojson hack (dbt-labs#572) * bump to 090a5 * update changelog * bump (dbt-labs#574) * 090 docs (dbt-labs#575) * 090 docs * Update CHANGELOG.md * Update CHANGELOG.md * Raise CompilationException on duplicate model (dbt-labs#568) * Raise CompilationException on duplicate model Extend tests * Ignore disabled models in parse_sql_nodes Extend tests for duplicate model * Fix preexisting models * Use double quotes consistently Rename model-1 to model-disabled * Fix unit tests * Raise exception on duplicate model across packages Extend tests * Make run_started_at timezone aware (dbt-labs#553) (dbt-labs#556) * Make run_started_at timezone aware Set run_started_at timezone to UTC Enable timezone change in models Extend requirements Extend tests * Address comments from code review Create modules namespace to context Move pytz to modules Add new dependencies to setup.py * Add warning for missing constraints. Fixes dbt-labs#592 (dbt-labs#600) * Add warning for missing constraints. Fixes dbt-labs#592 * fix unit tests * fix schema tests used in, or defined in packages (dbt-labs#599) * fix schema tests used in, or defined in packages * don't hardcode dbt test namespace * fix/actually run tests * rm junk * run hooks in correct order, fixes dbt-labs#590 (dbt-labs#601) * run hooks in correct order, fixes dbt-labs#590 * add tests * fix tests * pep8 * change req for snowflake to fix crypto install issue (dbt-labs#612) From cffi callback <function _verify_callback at 0x06BF2978>: Traceback (most recent call last): File "c:\projects\dbt\.tox\pywin\lib\site-packages\OpenSSL\SSL.py", line 313, in wrapper _lib.X509_up_ref(x509) AttributeError: module 'lib' has no attribute 'X509_up_ref' From cffi callback <function _verify_callback at 0x06B8CF60>: * Update python version in Makefile from 3.5 to 3.6 (dbt-labs#613) * Fix/snowflake custom schema (dbt-labs#626) * Fixes already opened transaction issue For dbt-labs#602 * Fixes dbt-labs#621 * Create schema in archival flow (dbt-labs#625) * Fix for pre-hooks outside of transactions (dbt-labs#623) * Fix for pre-hooks outside of transactions dbt-labs#576 * improve tests * Fixes already opened transaction issue (dbt-labs#622) For dbt-labs#602 * Accept string for postgres port number (dbt-labs#583) (dbt-labs#624) * Accept string for postgres port number (dbt-labs#583) * s/str/basestring/g * print correct run time (include hooks) (dbt-labs#607) * add support for late binding views (Redshift) (dbt-labs#614) * add support for late binding views (Redshift) * fix bind logic * wip for get_columns_in_table * fix get_columns_in_table * fix for default value in bind config * pep8 * skip tests that depend on nonexistent or disabled models (dbt-labs#617) * skip tests that depend on nonexistent or disabled models * pep8, Fixes dbt-labs#616 * refactor * fix for adapter macro called within packages (dbt-labs#630) * fix for adapter macro called within packages * better error message * Update CHANGELOG.md (dbt-labs#632) * Update CHANGELOG.md * Update CHANGELOG.md * Bump version: 0.9.0 → 0.9.1 * more helpful exception for registry funcs * Rework deps to support local & git * pylint and cleanup * make modules directory first * Refactor registry client for cleanliness and better error handling * init converter script * create modules directory only if non-existent * Only check the hub registry for registry packages * Incorporate changes from Drew's branch Diff of original changes: https://github.com/fishtown-analytics/dbt/pull/591/files * lint * include a portion of the actual name in destination directory * Install dependencies using actual name; better exceptions * Error if two dependencies have same name * Process dependencies one level at a time Included in this change is a refactor of the deps run function for clarity. Also I changed the resolve_version function to update the object in place. I prefer the immutability of this function as it was, but the rest of the code doesn't really operate that way. And I ran into some bugs due to this discrepancy. * update var name * Provide support for repositories in project yml * Download files in a temp directory The downloads directory causes problems with the run command because this directory is not a dbt project. Need to download it elsewhere. * pin some versions * pep8-ify * some PR feedback changes around logging * PR feedback round 2 * Fix for redshift varchar bug (dbt-labs#647) * Fix for redshift varchar bug * pep8 on a sql string, smh * Set global variable overrides on the command line with --vars (dbt-labs#640) * Set global variable overrides on the command line with --vars * pep8 * integration tests for cli vars * Seed rewrite (dbt-labs#618) * loader for seed data files * Functioning rework of seed task * Make CompilerRunner fns private and impl. SeedRunner.compile Trying to distinguish between the public/private interface for this class. And the SeedRunner doesn't need the functionality in the compile function, it just needs a compile function to exist for use in the compilation process. * Test changes and fixes * make the DB setup script usable locally * convert simple copy test to use seeed * Fixes to get Snowflake working * New seed flag and make it non-destructive by default * Convert update SQL script to another seed * cleanup * implement bigquery csv load * context handling of StringIO * Better typing * strip seeder and csvkit dependency * update bigquery to use new data typing and to fix unicode issue * update seed test * fix abstract functions in base adapter * support time type * try pinning crypto, pyopenssl versions * remove unnecessary version pins * insert all at once, rather than one query per row * do not quote field names on creation * bad * quiet down parsedatetime logger * pep8 * UI updates + node conformity for seed nodes * add seed to list of resource types, cleanup * show option for CSVs * typo * pep8 * move agate import to avoid strange warnings * deprecation warning for --drop-existing * quote column names in seed files * revert quoting change (breaks Snowflake). Hush warnings * use hub url * Show installed version, silence semver regex warnings * sort versions to make tests deterministic. Prefer higher versions * pep8, fix comparison functions for py3 * make compare function return value in {-1, 0, 1} * fix for deleting git dirs on windows? * use system client rmdir instead of shutil directly * debug logging to identify appveyor issue * less restrictive error retry * rm debug logging * s/version/revision for git packages * more s/version/revision, deprecation cleanup * remove unused semver codepath * plus symlinks!!! * get rid of reference to removed function
joellabes · Feb 27, 2018 · 5fbcd12 · 5fbcd12
1 parent 6783966
commit 5fbcd12
Show file tree

Hide file tree

Showing 13 changed files with 1,231 additions and 124 deletions.
diff --git a/converter.py b/converter.py
@@ -0,0 +1,73 @@
+#!/usr/bin/env python
+import json
+import yaml
+import sys
+import argparse
+from datetime import datetime, timezone
+import dbt.clients.registry as registry
+
+
+def yaml_type(fname):
+    with open(fname) as f:
+        return yaml.load(f)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--project", type=yaml_type, default="dbt_project.yml")
+    parser.add_argument("--namespace", required=True)
+    return parser.parse_args()
+
+
+def get_full_name(args):
+    return "{}/{}".format(args.namespace, args.project["name"])
+
+
+def init_project_in_packages(args, packages):
+    full_name = get_full_name(args)
+    if full_name not in packages:
+        packages[full_name] = {
+            "name": args.project["name"],
+            "namespace": args.namespace,
+            "latest": args.project["version"],
+            "assets": {},
+            "versions": {},
+        }
+    return packages[full_name]
+
+
+def add_version_to_package(args, project_json):
+    project_json["versions"][args.project["version"]] = {
+        "id": "{}/{}".format(get_full_name(args), args.project["version"]),
+        "name": args.project["name"],
+        "version": args.project["version"],
+        "description": "",
+        "published_at": datetime.now(timezone.utc).astimezone().isoformat(),
+        "packages": args.project.get("packages") or [],
+        "works_with": [],
+        "_source": {
+            "type": "github",
+            "url": "",
+            "readme": "",
+        },
+        "downloads": {
+            "tarball": "",
+            "format": "tgz",
+            "sha1": "",
+        },
+    }
+
+
+def main():
+    args = parse_args()
+    packages = registry.packages()
+    project_json = init_project_in_packages(args, packages)
+    if args.project["version"] in project_json["versions"]:
+        raise Exception("Version {} already in packages JSON"
+                        .format(args.project["version"]),
+                        file=sys.stderr)
+    add_version_to_package(args, project_json)
+    print(json.dumps(packages, indent=2))
+
+if __name__ == "__main__":
+    main()
diff --git a/dbt/adapters/bigquery.py b/dbt/adapters/bigquery.py
@@ -372,6 +372,12 @@ def get_columns_in_table(cls, profile, schema_name, table_name,
         raise dbt.exceptions.NotImplementedException(
             '`get_columns_in_table` is not implemented for this adapter!')
 
+    @classmethod
+    def get_columns_in_table(cls, profile, schema_name, table_name,
+                             model_name=None):
+        raise dbt.exceptions.NotImplementedException(
+            '`get_columns_in_table` is not implemented for this adapter!')
+
     @classmethod
     def check_schema_exists(cls, profile, schema, model_name=None):
         conn = cls.get_connection(profile, model_name)

diff --git a/dbt/clients/git.py b/dbt/clients/git.py
@@ -1,3 +1,4 @@
+import re
 import os.path
 
 from dbt.clients.system import run_cmd, rmdir
@@ -29,7 +30,7 @@ def checkout(cwd, repo, branch=None):
     if branch is None:
         branch = 'master'
 
-    logger.info('  Checking out branch {}.'.format(branch))
+    logger.debug('  Checking out branch {}.'.format(branch))
 
     run_cmd(cwd, ['git', 'remote', 'set-branches', 'origin', branch])
     run_cmd(cwd, ['git', 'fetch', '--tags', '--depth', '1', 'origin', branch])
@@ -59,3 +60,32 @@ def get_current_sha(cwd):
 
 def remove_remote(cwd):
     return run_cmd(cwd, ['git', 'remote', 'rm', 'origin'])
+
+
+def clone_and_checkout(repo, cwd, dirname=None, remove_git_dir=False,
+                       branch=None):
+    _, err = clone(repo, cwd, dirname=dirname, remove_git_dir=remove_git_dir)
+    exists = re.match("fatal: destination path '(.+)' already exists",
+                      err.decode('utf-8'))
+    directory = None
+    start_sha = None
+    if exists:
+        directory = exists.group(1)
+        logger.debug('Updating existing dependency %s.', directory)
+    else:
+        matches = re.match("Cloning into '(.+)'", err.decode('utf-8'))
+        directory = matches.group(1)
+        logger.debug('Pulling new dependency %s.', directory)
+    full_path = os.path.join(cwd, directory)
+    start_sha = get_current_sha(full_path)
+    checkout(full_path, repo, branch)
+    end_sha = get_current_sha(full_path)
+    if exists:
+        if start_sha == end_sha:
+            logger.debug('  Already at %s, nothing to do.', start_sha[:7])
+        else:
+            logger.debug('  Updated checkout from %s to %s.',
+                         start_sha[:7], end_sha[:7])
+    else:
+        logger.debug('  Checked out at %s.', end_sha[:7])
+    return directory
diff --git a/dbt/clients/registry.py b/dbt/clients/registry.py
@@ -0,0 +1,61 @@
+from functools import wraps
+import six
+import requests
+from dbt.exceptions import RegistryException
+from dbt.utils import memoized
+import os
+
+if os.getenv('DBT_PACKAGE_HUB_URL'):
+    DEFAULT_REGISTRY_BASE_URL = os.getenv('DBT_PACKAGE_HUB_URL')
+else:
+    DEFAULT_REGISTRY_BASE_URL = 'https://hub.getdbt.com/'
+
+
+def _get_url(url, registry_base_url=None):
+    if registry_base_url is None:
+        registry_base_url = DEFAULT_REGISTRY_BASE_URL
+
+    return '{}{}'.format(registry_base_url, url)
+
+
+def _wrap_exceptions(fn):
+    @wraps(fn)
+    def wrapper(*args, **kwargs):
+        try:
+            return fn(*args, **kwargs)
+        except requests.exceptions.ConnectionError as e:
+            six.raise_from(
+                RegistryException('Unable to connect to registry hub'), e)
+    return wrapper
+
+
+@_wrap_exceptions
+def _get(path, registry_base_url=None):
+    url = _get_url(path, registry_base_url)
+    resp = requests.get(url)
+    resp.raise_for_status()
+    return resp.json()
+
+
+def index(registry_base_url=None):
+    return _get('api/v1/index.json', registry_base_url)
+
+
+index_cached = memoized(index)
+
+
+def packages(registry_base_url=None):
+    return _get('api/v1/packages.json', registry_base_url)
+
+
+def package(name, registry_base_url=None):
+    return _get('api/v1/{}.json'.format(name), registry_base_url)
+
+
+def package_version(name, version, registry_base_url=None):
+    return _get('api/v1/{}/{}.json'.format(name, version), registry_base_url)
+
+
+def get_available_versions(name):
+    response = package(name)
+    return list(response['versions'])
diff --git a/dbt/clients/system.py b/dbt/clients/system.py
@@ -5,8 +5,12 @@
 import shutil
 import subprocess
 import sys
+import tarfile
+import requests
+import stat
 
 import dbt.compat
+import dbt.exceptions
 
 from dbt.logger import GLOBAL_LOGGER as logger
 
@@ -92,19 +96,61 @@ def make_file(path, contents='', overwrite=False):
     return False
 
 
+def make_symlink(source, link_path):
+    """
+    Create a symlink at `link_path` referring to `source`.
+    """
+    if not supports_symlinks():
+        dbt.exceptions.system_error('create a symbolic link')
+
+    return os.symlink(source, link_path)
+
+
+def supports_symlinks():
+    return getattr(os, "symlink", None) is not None
+
+
 def write_file(path, contents=''):
     make_directory(os.path.dirname(path))
     dbt.compat.write_file(path, contents)
 
     return True
 
 
+def _windows_rmdir_readonly(func, path, exc):
+    exception_val = exc[1]
+    if exception_val.errno == errno.EACCES:
+        os.chmod(path, stat.S_IWUSR)
+        func(path)
+    else:
+        raise
+
+
 def rmdir(path):
     """
-    Make a file at `path` assuming that the directory it resides in already
-    exists. The file is saved with contents `contents`
+    Recursively deletes a directory. Includes an error handler to retry with
+    different permissions on Windows. Otherwise, removing directories (eg.
+    cloned via git) can cause rmtree to throw a PermissionError exception
     """
-    return shutil.rmtree(path)
+    logger.debug("DEBUG** Window rmdir sys.platform: {}".format(sys.platform))
+    if sys.platform == 'win32':
+        onerror = _windows_rmdir_readonly
+    else:
+        onerror = None
+
+    return shutil.rmtree(path, onerror=onerror)
+
+
+def remove_file(path):
+    return os.remove(path)
+
+
+def path_exists(path):
+    return os.path.lexists(path)
+
+
+def path_is_symlink(path):
+    return os.path.islink(path)
 
 
 def open_dir_cmd():
@@ -133,3 +179,27 @@ def run_cmd(cwd, cmd):
     logger.debug('STDERR: "{}"'.format(err))
 
     return out, err
+
+
+def download(url, path):
+    response = requests.get(url)
+    with open(path, 'wb') as handle:
+        for block in response.iter_content(1024*64):
+            handle.write(block)
+
+
+def rename(from_path, to_path, force=False):
+    if os.path.exists(to_path) and force:
+        rmdir(to_path)
+    os.rename(from_path, to_path)
+
+
+def untar_package(tar_path, dest_dir, rename_to=None):
+    tar_dir_name = None
+    with tarfile.open(tar_path, 'r') as tarball:
+        tarball.extractall(dest_dir)
+        tar_dir_name = os.path.commonprefix(tarball.getnames())
+    if rename_to:
+        downloaded_path = os.path.join(dest_dir, tar_dir_name)
+        desired_path = os.path.join(dest_dir, rename_to)
+        dbt.clients.system.rename(downloaded_path, desired_path, force=True)
diff --git a/dbt/context/common.py b/dbt/context/common.py
@@ -1,5 +1,6 @@
 import json
 import os
+import pytz
 import voluptuous
 
 from dbt.adapters.factory import get_adapter

diff --git a/dbt/deprecations.py b/dbt/deprecations.py
@@ -11,21 +11,19 @@ def show(self, *args, **kwargs):
             logger.info("* Deprecation Warning: {}\n".format(desc))
             active_deprecations.add(self.name)
 
-# Leaving this as an example. Make sure to add new ones to deprecations_list
-#       - Connor
-#
-# class DBTRunTargetDeprecation(DBTDeprecation):
-#     name = 'run-target'
-#     description = """profiles.yml configuration option 'run-target' is
-#     deprecated. Please use 'target' instead. The 'run-target' option will be
-#     removed (in favor of 'target') in DBT version 0.7.0"""
+
+class DBTRepositoriesDeprecation(DBTDeprecation):
+    name = "repositories"
+    description = """dbt_project.yml configuration option 'repositories' is
+    deprecated. Please use 'packages' instead. The 'repositories' option will
+    be removed in a later version of DBT."""
 
 
 class SeedDropExistingDeprecation(DBTDeprecation):
     name = 'drop-existing'
-    description = """The --drop-existing argument has been deprecated. Please
-  use --full-refresh instead. The --drop-existing option will be removed in a
-  future version of dbt."""
+    description = """The --drop-existing argument to `dbt seed` has been
+  deprecated. Please use --full-refresh instead. The --drop-existing option
+  will be removed in a future version of dbt."""
 
 
 def warn(name, *args, **kwargs):
@@ -44,6 +42,7 @@ def warn(name, *args, **kwargs):
 active_deprecations = set()
 
 deprecations_list = [
+    DBTRepositoriesDeprecation(),
     SeedDropExistingDeprecation()
 ]