Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cd: create a gha that generate an index for each query in the src/sql folder #5

Closed
wants to merge 13 commits into from
Closed

cd: create a gha that generate an index for each query in the src/sql folder #5

wants to merge 13 commits into from

Commits on Mar 25, 2024

  1. cd:create a gha that can generate an index for 

    every query in the src/sql folder. The gha will
check if the version of IDC is out of date. If
so, the queries will be updated, and run with
bigquery to generate csv and parquet files in
root directory. The cmake lists file is edited
to pick up the csv and parquet files generated
by bq and package them. Moreover gitignore is
edited to have a pattern for ignoring service
account files. Lastly, if queries are updated,
a pull request will be created to update the queries
to be in sync with latest idc release.
    @vkt1414 @jcfr
    vkt1414 authored and jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    df0c196 View commit details
    Browse the repository at this point in the history

  2. enh: re-organize sql queries and scripts used for gha 

    a helper python script is referred and run in gha to generate
indices
use static methods for functions that do not
rely on initialization of a class
sql queries are reverted back to use idc_current
but explicit versioned queries are uploaded as gha artifacts
pylint is configured to lint scripts folder as well
    @vkt1414 @jcfr
    vkt1414 authored and jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    dc81122 View commit details
    Browse the repository at this point in the history

  3. enh: add build dependencies directly to pyproject.toml 

    call the index generator from cmake lists
started using a single function to generate indices
instead of two taking boolean inputs whether or not
to generate csv, parquet files. Currently hard
coded to generate both formats and not fully using the
boolean capabilities yet with cmake.
Queries are no longer needed to be attached as gha artifacts
    @vkt1414 @jcfr
    vkt1414 authored and jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    9933808 View commit details
    Browse the repository at this point in the history

  4. enh: include gcp auth step to ci, remove pypi 3.10

    @vkt1414 @jcfr
    vkt1414 authored and jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    67f1e30 View commit details
    Browse the repository at this point in the history

  5. enh: Generate only csv for now

    @vkt1414 @jcfr
    vkt1414 authored and jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    fc64bfd View commit details
    Browse the repository at this point in the history

  6. fix: generate only csv for now

    @jcfr
    Vamsi Thiriveedhi authored and jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    52954f1 View commit details
    Browse the repository at this point in the history

  7. enh: use csv index only for now 

    fix minor formatting errors
    @vkt1414 @jcfr
    vkt1414 authored and jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    b2a5d45 View commit details
    Browse the repository at this point in the history

  8. style: Revert unneeded changes from "cd.yml"

    @jcfr
    jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    17c9e13 View commit details
    Browse the repository at this point in the history

  9. style: fix indent in CMakeLists.txt

    @jcfr
    jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    e002336 View commit details
    Browse the repository at this point in the history

  10. style: Tweak IDCIndexDataManager docstring formatting

    @jcfr
    jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    2503daa View commit details
    Browse the repository at this point in the history

  11. feat: Update idc_index_data_manager.py to accept arguments

    @jcfr
    jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    985c9c8 View commit details
    Browse the repository at this point in the history

  12. feat: Add IDC_INDEX_DATA_GENERATE_PARQUET option

    @jcfr
    jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    3324ffe View commit details
    Browse the repository at this point in the history

  13. feat: Add IDC_INDEX_DATA_GENERATE_CSV_ARCHIVE option (enabled by defa… 

    …ult)
    @jcfr
    jcfr committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    0ba0527 View commit details
    Browse the repository at this point in the history