Support graphql feature

[jetersen]

Description
Support GraphQL API queries, GraphQL query explorer
Example query

query {
  group(fullPath: "gitlab-org") {
    avatarUrl
    description
    fullName
    fullPath
    id
    lfsEnabled
    name
    parent {
      name
    }
    path
    requestAccessEnabled
    userPermissions {
      readGroup
    }
    visibility
    webUrl
  }
  project(fullPath: "gitlab-org/gitlab-ce") {
    archived
    avatarUrl
    containerRegistryEnabled
    createdAt
    description
    forksCount
    fullPath
    group {
      name
    }
    httpUrlToRepo
    id
    importStatus
    issuesEnabled
    issue(iid: 1){
      author {
        name
        username
        webUrl
        avatarUrl
      }
      title
      description
      webUrl
    }
    issues(first: 5) {
      edges {
        node {
          author {
            name
            username
            webUrl
            avatarUrl
          }
          title
          description
          webUrl
        }
      }
      pageInfo {
        hasNextPage
        endCursor
      }
    }
    jobsEnabled
    lastActivityAt
    lfsEnabled
    mergeRequestsEnabled
    mergeRequest(iid: 1) {
      webUrl
    }
    mergeRequests(last: 5) {
      edges {
        node {
          allowCollaboration
          createdAt
          defaultMergeCommitMessage
          description
          diffHeadSha
          downvotes
          forceRemoveSourceBranch
          headPipeline {
            coverage
          }
          id
          iid
          inProgressMergeCommitSha
          mergeCommitSha
          mergeError
          mergeOngoing
          mergeStatus
          mergeWhenPipelineSucceeds
          mergeableDiscussionsState
          pipelines {
            edges {
              node {
                coverage
              }
            }
          }
          project {
            webUrl
          }
          projectId
          rebaseCommitSha
          rebaseInProgress
          shouldBeRebased
          shouldRemoveSourceBranch
          sourceBranch
          sourceBranchExists
          sourceProject {
            webUrl
          }
          sourceProjectId
          state
          subscribed
          targetBranch
          targetProject {
            webUrl
          }
          targetProjectId
          title
          updatedAt
          upvotes
          userNotesCount
          userPermissions {
            adminMergeRequest
            cherryPickOnCurrentMergeRequest
            createNote
            pushToSourceBranch
            readMergeRequest
            removeSourceBranch
            revertOnCurrentMergeRequest
            updateMergeRequest
          }
          webUrl
          workInProgress
        }
      }
    }
    mergeRequestsFfOnlyEnabled
    name
    nameWithNamespace
    namespace {
      name
    }
    onlyAllowMergeIfAllDiscussionsAreResolved
    onlyAllowMergeIfPipelineSucceeds
    openIssuesCount
    path
    pipelines(first: 5) {
      edges {
        node {
          committedAt
          createdAt
          startedAt
          finishedAt
        }
      } 
      pageInfo {
        hasNextPage
        endCursor
      }
    }
    printingMergeRequestLinkEnabled
    publicJobs
    requestAccessEnabled
    sharedRunnersEnabled
    snippetsEnabled
    sshUrlToRepo
    starCount
    tagList
    userPermissions {
      adminProject
      adminRemoteMirror
      adminWiki
      archiveProject
      changeNamespace
      changeVisibilityLevel
      createDeployment
      createDesign
      createIssue
      createLabel
      createMergeRequestFrom
      createMergeRequestIn
      createPages
      createPipeline
      createPipelineSchedule
      createProjectSnippet
      createWiki
      destroyDesign
      destroyPages
      destroyWiki
      downloadCode
      downloadWikiCode
      forkProject
      pushCode
      pushToDeleteProtectedBranch
      readCommitStatus
      readCycleAnalytics
      readPagesContent
      readProject
      readProjectMember
      readWiki
      removeForkProject
      removePages
      removeProject
      renameProject
      requestAccess
      updatePages
      updateWiki
      uploadFile
    }
    visibility
    webUrl
    wikiEnabled
  }
}

Proposal
Would be great to already start implementing the use case for using Gitlab's new GraphQL API

[jetersen]

Updated with an example query 😅

[jdalrymple]

Would we just expose a GraphQL model which would be initialized with the users settings, and then it would have one function "query" which would take in a graphql query like above?

[jetersen]

I think that would be wonderful 😅

[jetersen]

The query should of course support taking in variables as well 😓

[jdalrymple]

Not sure exactly how it would all work, but i guess we could brain storm some things here!

[jetersen]

The easiest thing would be pull in a GraphQL client? No need to rewrite everything.
Though not that it requires that much code: https://www.npmjs.com/package/graphql-request
https://github.com/prisma/graphql-request/blob/master/src/index.ts

[jdalrymple]

Keeping things simple, I like it. Easy to include the graphQL client. This could probably be done relatively quickly...

[jdalrymple]

I was thinking about this today. What would be the best way to expose this? As a service? so you'd import it like:

import { GitLabQL } from 'gitlab';

[jetersen]

GitLabQL seems reasonable, I don't think it should be in the standard bundle :)

[jdalrymple]

Would this be something only available to node users? As in though this be its own package @gitbeaker/graphql ?

[jetersen]

This would be beneficial to both node and browser :)

[jdalrymple]

Can you make requests to the graphql api through standard http requests? or does it require a graphql requester that supports browser and node?

[jetersen]

you can do with normal http requests it is just post method.
Though there is some special logic handling queries and pagination.

[jdalrymple]

you can do with normal http requests it is just post method.
Though there is some special logic handling queries and pagination.

Any examples of that special logic. I could try to incorporate it into the Got/Ky Requesters

[jdalrymple]

Rereading the docs, this is very doable, the only issue is determining how the pagination would work 🤔 Would there be any? Or would the user need to handle that themselves

[ThePlenkov]

It would be awesome to have it. Just now face the issue: i want to find upstream pipeline for a child pipeline. REST API doesn't have this reference (https://docs.gitlab.com/ee/api/pipelines.html#get-a-single-pipeline) and GraphQL does https://docs.gitlab.com/ee/api/graphql/reference/#pipeline. We can simply see how much more data we have in GraphQL version. So it could be nice to use also gitbeaker to acess those resources =)

[jdalrymple]

I definitely want to get to this, just need to find the time 😞

[ThePlenkov]

@jdalrymple i can imagine - I try now to create generated typescript types out of gitlab graph ql scema.

Menwhile one more highlist - is not possible to use only graphql. I just found the case : to find MR by commit we need this api, https://docs.gitlab.com/ee/api/commits.html#list-merge-requests-associated-with-a-commit but there is not navigation to commit from pipeline in graphql, only sha. So now I need to build a hybrid task to combine two kind f calls to get the original MR by a child pipeline =)

[jdalrymple]

Ouf that does not seem pleasant :s

[ThePlenkov]

Created a simple type generator: https://github.com/theplenkov-npm/gitlab-graphql-types. May be someone finds it useful =)

[trevor-vaughan]

I was looking for GraphQL support in the @gitbeaker library and would like to suggest pulling in the graphql-request package and wrapping it via a simple wrapper.

In theory, something like this would work (javascript pseudo-code-ish not typescript but 🤷)

const { graphQLClient } = require('graphql-request')
# Making remove-deprecated an option would be great
const graphql = new GraphQLClient(this.url + '/api/graphql?remove_deprecated=true')

graphql.setHeaders({  authorization: `Bearer ${gitlab_token}` })

The lib would then need to accept whatever the request() method from graphql-request can accept. I think that a direct passthrough would probably work for most users.

The request() method doesn't handle paging natively, so it would be great if the library could easily handle page processing. Quick example (untested):

async graphqlQuery(graphqlClient, query, vars, endpoint) {
  let result = await graphqlClient.request(query, vars)

  let toProcess = result[endpoint]
  let returnVal = toProcess.nodes

  let pageInfo = toProcess.pageInfo
  let curPage = pageInfo.endCursor
  if ( pageInfo.hasNextPage ) {
     curPage = pageInfo.endCursor
    result = await graphqlClient.request(query, vars)
    returnVal = returnVal.concat(result[endpoint].nodes)
    pageInfo = result[endpoint].pageInfo
  }

  return returnVal
}

[jdalrymple]

Yea, i was thinking of making another lib @gitbeaker/gql or something of the sort that would provide this support. Basically doing as you mentioned. There would be some changes since the core library handles pagination atm and that would need to be abstracted out, or the responses would need to be structured. Very doable though.

[trevor-vaughan]

Yeah, that's why I was figuring that it would end up just being a wrapper around graphql-request honestly.

Would a @gitbeaker/gql library just add on an additional endpoint? That would be pretty reasonable.

I think that the example I provided would automatically handle paging for you pretty seamlessly (it works for me anyway).

[trevor-vaughan]

🤔 Alternatively, you may want to make it part of the core library so that you can start moving some items over to use it natively.

The biggest issue is that GitLab currently doesn't have parity between the GraphQL and REST interfaces in both directions. (It's making m e crazy)

[jdalrymple]

Trust me, the fact that its inconsistent APIs for both REST and GraphQL is the bane of my existence haha.

Since the graphql implementation requires one to pass the query, depends on a custom request lib and isnt at all connected to the rest endpoints, it is unlikely that i could just add it to the Core or another library to integrate it well. This would most likely have to be its own standalone thing. Ill see if i can throw together something over the weekend and follow up!

[jdalrymple]

Actually, upon looking at the code a bit more, you may be right. It could work just as an additional endpoint. Id have to tinker a bit i think.

[trevor-vaughan]

I also did a little tinkering and I think the biggest issue is if you want to make the paging "easy".

For instance, you'd have to process the statement to ensure that the appropriate page info material was attached to any node entries. That's honestly all of the magic that I think is needed though for the general case.

I don't think that's a huge lift but it's certainly annoying. You might be able to tap into the gql template literal to make it easier.

For a first cut, just literally exposing the underlying hooks (maybe call it an experimental feature) makes sense to me.

[trevor-vaughan]

🤔 Actually, looking at the original post query, you can end up with a bunch of endpoints with paging.

Well, you could possibly make people add the paging hooks if they want them (maybe you only want the first page). Then, you could remember those hook locations and post-process the paging for them based on where they indicated that they wanted the paging to occur.

In theory, you could do all of this in parallel but that might drop some excessive load on the target server.

However, if you take this approach, making folks deal with paging on their own for round 1 and then adding auto-paging later would not be a breaking change.

[jdalrymple]

Not a bad idea, releasing it as an experimental feature provides basic support without prod expectations

[jdalrymple]

Gonna jump on this one next!

[trevor-vaughan]

Did some more playing with this recently and have some thoughts:

Common Fragments

I would make a spot for common fragments to be used. By default, I would add a PageInfo fragment something like:

fragment pInfo on PageInfo {
  hasNextPage
  endCursor
}

Folks can use it via the gql template something like:

let query = gql`
  ${gitLab.GraphQL.fragments.pageInfo}

  query {
    #query stuff
    edges {
      node {
        something
      }
    }
    pageInfo {
      ...pInfo
    }
`

Auto-Paging

Unfortunately, auto-paging continues to be a pain. You can have multiple locations where paging can happen in a given query so you end up with a set of items that is pretty easy to page locally but creating variables to handle it automatically is really difficult.

Essentially, you can start with an after: parameter on something like issues and then feed that back in as a variable. Unfortunately, the dynamic nature of queries really does make it difficult to automatically figure out exactly where to page.

That said, it's pretty much as simple as this for a single-point query (using graphql-request). Note: I used nodes instead of edges as a shortcut since 99% of the time we don't care about the edge information.

const _ = require('lodash')

let query = gql`
  ${gitLab.GraphQL.fragments.pageInfo}

  query($curPage: String!) {
    project(fullPath: "my/project/path") {
      issues(after: $curPage) {
        nodes {
          id
        }
        pageInfo {
          ...pInfo
        }
      }
  }
`

let vars = {
  curPage: ''
}

let result = await gitLab.GraphQL.client.request(query, vars)

let pageInfo = result.project.issues.pageInfo
do {
  vars.curPage = pageInfo.endCursor

  // Need that deep merge capability
  _.merge(result, await gitlab.GraphQL.client.request(query, vars)
} while(pageInfo.hasNextPage)

[jdalrymple]

I noticed octokit doesnt use the graphql library for their graphql support. Would this be a possible avenue to avoid the additional dep?

[trevor-vaughan]

Yeah, there really isn't much to it, and you've already got a solid set of calls under the hood so it's probably fine.

[trevor-vaughan]

@jdalrymple In case you want to prod GitLab with us https://gitlab.com/gitlab-org/gitlab/-/issues/415519#note_1639296965