lab project list offers only incomplete results

[alerque]

Running lab project list I expect to find all my user's projects, but instead I get a small sampling of them. There does not appear to be any pagination mechanism, and adding the -a flag does nothing, I get the same truncated list.

Adding a search term does pull up results not in the list, but it only searches by matches in the project name.

I'm trying to get the list of projects in a group, but this seems impossible. I could grep the results for the group name if it returned a full list, but it does not. Searching doesn't search on the group name, so that doesn't help.

[zaquestion]

@alerque, pagination works by passing a number as the last argument, with with out without a search. Similar is true in the other list commands.

By default only your projects are listed, -a makes includes the rest. Right now the command is hard code to only return 10 results per page, I'm planning to make that configurable like in other list commands in the next release. Let me know if any of that helps

[zaquestion]

Oh weird, my sleepy eyes over read your bit about groups. I'm sure theres something more we can do to specifically make that better. I'll check the gitlab API when I'm awake

[alerque]

Okay pagination does work, I just didn't understand how to use it. And I would add that it wasn't at all clear from the help pages. Perhaps #147 could use some attention on that front.

That does changes this issue a bit, but it still doesn't accomplish what I need. Scripting something that gives paginated output is really complicated. I'm trying to write scripts that check out all projects in a group, any the only way I see to do it while a loop that iterates over an incrementing integer and test until it fails, then combine the results. This is tedious and brittle scripting for something that should be one command. If the API must be iterated to get all results, this is something lab should do internally. Doing a list or search it should at least be possible to request all results.

Additionally being able to list or search inside a group specifically would be useful.

[zaquestion]

@alerque Thanks for the great feedback here. I completely agree there needs to be an easier way to work with all results (pages). I haven't forgotten about this issue, particularly the bit about groups.

Any suggestions on syntax are appreciated.

Some ideas:

• change the -a meaning to paginate and return all pages, find a new flag for differentiating user projects from all projects.
• accept a range for the page argument where 1- would mean 1 to the end
• I still haven't looked at the API to see if group seaches are supported, but lab can easily do this somehow.
  • accept a -g for the group
  • recognize a / in the search and do a prefix search instead.

[zaquestion]

Also forgot to underscore, I love the idea of being able to clone an entire group. I'll look at implementing that for lab clone too

[zaquestion]

After playing around with this a bit, I'm thinking the way to handle getting all the results is to use -n to set the number of results and paginate if that number is above the max results per page. Currently in other commands we only return the max per page, but that doesn't seem very useful or intuitive. I'm planning to make these changes across all list sub commands.

Also remove --all, -a to reduce confusion and follow the gitlab API flags closer to their filtering options.

	projectListCmd.Flags().BoolVarP(&projectListConfig.Owned, "mine", "m", false, "limit by your projects")
	projectListCmd.Flags().BoolVar(&projectListConfig.Membership, "member", false, "limit by projects which you are a member")
	projectListCmd.Flags().BoolVar(&projectListConfig.Starred, "starred", false, "limit by your starred projects")

Regarding groups looks like we can use the Groups API to list all projects in a group. https://docs.gitlab.com/ee/api/groups.html#list-a-group-s-projects

I'm currently thinking a flag for lab project list and a sub command for lab clone (lab clone group)

[alerque]

From a CLI perspective paginated results are not useful. Ever. Shells have ways of handling that; if output is too much to look at one uses commands like head or less. Take for example the output if git log: by default it returns the entire repository history. It does have -<N> argument to only process the most recent N entries, but this is not the default output.

The main reason for lab ever returning paginated results would be the overhead of Gitlab generating the results and the overhead of downloading them. This might make sense in the case of looking at project activity, but I still think the default could be to not return paginated results at all. If the Gitlab API will only send paginated results, lab should still default to iterating over all pages and returning the full results. Then if there is a numeric argument to limit results it could optimize it's requests and only fetch enough data to return that subset.

I agree the -a option is confusing and should probably be the default, with a -u or similar flag to restrict to a only a user's owned repos which would nicely compliment a -g flag to restrict to a specific group.

[zaquestion]

Here's what I current have going. Agree on paginated results not being very useful and upon reflection the user shouldn't really have to think about there being a paginated API.

$ lab project list -n 200 | wc -l
200

$ lab project list -n 300 | wc -l
300

$ time lab project list -a | wc -l
2443

real    0m25.653s
user    0m0.398s
sys     0m0.140s

At the moment I'm planning to bump the default results up to 100 (max for 1 page). But since its easy for instances to have enough projects for the operation to take a long time for all projects (think gitlab.com) I'm still thinking you should have to be explicit.

I'm going to get this implemented for the other list commands. Regarding group operations, I'm thinking about just supporting the various options through a new lab group command. Then poking some -g in other commands where appropriate.

[alerque]

That sounds reasonable to me. Having to explicitly set a higher limit makes some sense in the case of making a bunch of API queries. I did hack the limits off a build for myself and was surprised to find how slow the Gitlab API responded to the request. For just listing available project names it's pretty disappointing to wait that long when the full project list web pages load faster than that. It's bad enough that I'm going to have to cache the results locally. It's really convenient to get the list in my script, but I need to run the script dozens of times per day and 30 seconds a pop is unacceptable. Having the latest full project list is really nice, but it doesn't change minute to minute so much as day to day.

[zaquestion]

Partial fix implements the better listing described above: #195

[zaquestion]

@alerque lab should produce complete (fully paginated) list results when using -a or will return -n <num> results with v0.13.0. I've also created a followup ticket for better group operations at #206. With that in mind I'm going to close this one out, let me know if I missed anything!