Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRILL-7191 / DRILL-7026]: RM state blob persistence in Zookeeper and Integration of Distributed queue configuration with Planner #1762

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Commits on Apr 22, 2019

  1. Configuration menu
    Copy the full SHA
    097122e View commit details
    Browse the repository at this point in the history
  2. DRILL-7193: Integration changes of the Distributed RM queue configura…

    …tion with Simple Parallelizer.
    
    Refactor existing ZK based queue to accommodate new Distributed queue for RM.
    Refactor and rename the existing memory allocation utilities to ZKQueueMemoryAllocationUtilities and DefaultMemoryAllocationUtilities.
    Parallelizer code is changed to accommodate the memory adjustment for the operators during parallelization phase.
    With this change, there are 3 different implementation of SimpleParallelizer; they are ZKQueueParallelizer, DistributedQueueParallelizer and DefaultParallelizer which will be used by ZK based RM, Distributed RM and Non RM configuration.
    HanumathRao committed Apr 22, 2019
    Configuration menu
    Copy the full SHA
    7e71cd0 View commit details
    Browse the repository at this point in the history
  3. DRILL-7191: RM blobs persistence in Zookeeper for Distributed RM.

    UUID support for DrillbitEndpoint
    RMState Blobs definition, serialization and deserialization, Zookeeper client support for transactions
    ZookeeperPersistentTransactional Store and RMStateBlobManager to do updates under lock
    Protect running and waiting queries map in WorkerBee
    sohami authored and HanumathRao committed Apr 22, 2019
    Configuration menu
    Copy the full SHA
    1db65a2 View commit details
    Browse the repository at this point in the history
  4. DRILL-7193: Integration changes of the Distributed RM queue configura…

    …tion with Simple Parallelizer.
    
    Integration changes with new DistributedRM queue configuration.
    a) Remove the redundant NodeResource and merge the additional member functions with the NodeResources class.
    b) Added new UUID logic and selection of a queue based on the memory requirement during parallelization phase.
    c) Changed proto definitions to set the UUID of a drillbit.
    d) Implementation of new DrillNode Wrapper over DrillbitEndpoint to fix the equality comparisions between DrillbitEndpoints.
    HanumathRao committed Apr 22, 2019
    Configuration menu
    Copy the full SHA
    2b6a91a View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2019

  1. DRILL-7191: RM blobs persistence in Zookeeper for Distributed RM.

    Added stubs for QueryResourceManager exit and wait/cleanup thread
    Update MemoryCalculator to use DrillNode instead of DrillbitEndpoint
    Changes to support localbit resource registration to cluster state blob using DrillbitStatusListener
    Support ThrottledResourceManager via ResourceManagerBuilder
    Add some E2E tests and RMStateBlobs tests along with some bug fixes
    Fix TestRMConfigLoad tests to handle case where ZKQueues are explicitly enabled
    sohami committed Apr 23, 2019
    Configuration menu
    Copy the full SHA
    2800c57 View commit details
    Browse the repository at this point in the history
  2. DRILL-7193: Integration changes of the Distributed RM queue configura…

    …tion with Simple Parallelizer.
    
    Changes to set the memory allocation per operator in query profile.
    Addressing an memory minimization logic was not considering non-buffered operators.
    Handling error cases when memory requirements for buffered or non-buffered cannot be reduced.
    HanumathRao authored and sohami committed Apr 23, 2019
    Configuration menu
    Copy the full SHA
    1517a87 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2019

  1. Configuration menu
    Copy the full SHA
    a3f8b36 View commit details
    Browse the repository at this point in the history