Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Monorepo example #1

Open
jdinard opened this issue Jun 27, 2019 · 23 comments
Open

Monorepo example #1

jdinard opened this issue Jun 27, 2019 · 23 comments

Comments

@jdinard
Copy link

jdinard commented Jun 27, 2019

Hello,

Just trying to evaluate semaphore for a project with about 40 containerized microservices in a monorepo. The docs state that this is where monorepo examples live, but there don't seem to be any.

https://docs.semaphoreci.com/article/50-pipeline-yaml
@markoa
Copy link
Contributor

markoa commented Jun 27, 2019

Hi, sorry for non-direct link, the example is in the monorepo branch in this repo.

@jdinard
Copy link
Author

jdinard commented Jun 28, 2019

Thats awesome. Thank you.

Is there a way to define pipelines on a specific folder?

For example, our monorepo is structured like:

/root
/service1
/service2
/service3

We don't want to build/trigger tests for service2 when someone commits a change that only impacts files in service1.

@markoa
Copy link
Contributor

markoa commented Jun 28, 2019 via email

@philippks
Copy link

Any news or hint about this @markoa? we've exactly the same use case:

We don't want to build/trigger tests for service2 when someone commits a change that only impacts files in service1.

@markoa
Copy link
Contributor

markoa commented Jul 24, 2019 via email

@steffenmllr
Copy link

@markoa that would be great - I'm looking to move a complex monorepo from github actions (too slow) to semaphore

@schmijos
Copy link

I'm on this as well. On CircleCI I could comfortably set the working directory. Should I just cd here all the time?

@markoa
Copy link
Contributor

markoa commented Oct 21, 2019 via email

@markoa
Copy link
Contributor

markoa commented Oct 21, 2019 via email

@schmijos
Copy link

In a monorepo I need to change to subdirectories by myself (compared to CircleCI where the working directory is configurable). I think my previous concern is a non-issue.

@markoa
Copy link
Contributor

markoa commented Jan 9, 2020

Folks, I'm happy to let you know that we have shipped monorepo support on Semaphore.

Please see this documentation page to get started, and let me know your impressions when you get a chance.

@schmijos
Copy link

schmijos commented Jan 10, 2020 •
edited
Loading

Very cool how you keep listening to your customers!
Please also think about caching because the cache keys currently need to be isolated manually and I end up doing error prone things like this (actually I use cache has_key to prevent some errors):

- name: "Setup Customer Mobile App"
  commands:
    - cd web
    - cache restore mcweb-nodemodules-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock),mcweb-nodemodules-$SEMAPHORE_GIT_BRANCH,mcweb-nodemodules-master
    - yarn install
    - cache store mcweb-nodemodules-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock) node_modules
    - cd -
    - cd app
    - cache restore yarn-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock),yarn-$SEMAPHORE_GIT_BRANCH,yarn-master
    - cache restore mc-nodemodules-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock),mc-nodemodules-$SEMAPHORE_GIT_BRANCH,mc-nodemodules-master
    - yarn install
    - cache store yarn-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock) ~/.cache/yarn
    - cache store mc-nodemodules-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock) node_modules
…
- name: "Customer Mobile App Checks"
  …
    prologue:
      commands:
        - checkout
        - cd web
        - cache has_key mcweb-nodemodules-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock)
        - cache restore mcweb-nodemodules-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock)
        - cd -
        - cd app
        - cache has_key yarn-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock)
        - cache has_key mc-nodemodules-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock)
        - cache restore yarn-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock)
        - cache restore mc-nodemodules-$SEMAPHORE_GIT_BRANCH-$(checksum yarn.lock)
  …

I started with setups like these because I accidentally started overwriting vendor/bundle folders for multiple Ruby projects in the same monorepo when using the generic cache store.

@shiroyasha
Copy link

@schmijos Thank you for providing feedback about caching in monorepo contexts. We are designing improvements for monorepo caching.

One solution that we are investigating is to introduce namespaces into the cache, where each service could define its own namespace.

For example: cache store --namespace "mobile-app" and cache store --namespace "web-app".

Would this be a good solution to your project?

@schmijos
Copy link

@shiroyasha This would be exactly what we need ❤️

@LaloLoop
Copy link

Hello, I am having trouble with the documentation, as it states that we can change the agent per block defined in a monorepo. I have one defined in my project consisting of a python backend and iOS app. Whenever I try to set the iOS subfolder to use a different agent, I get the following error in my pipeline:

Error: [{"Schema does not allow additional properties.", "#/blocks/1/agent"}]

The documentation shows that this should be possible, is this actually available right now in the platform or do I need to activate something to enable the beta features?

I saw that I can enable the agent per task, but that's not really ideal, since the iOS block will have more tasks added (building and shipping).

Please let me know if I can provide you with more information, and thank you for adding this!

@LaloLoop LaloLoop mentioned this issue May 10, 2020
Merged
@shiroyasha
Copy link

@LaloLoop Thanks for reporting. It's a mistake in our documentation.

Agent can be defined on the task level, not on the block level. Instead of:

  - name: Test iOS client
    dependencies: []
    run:
      when: "change_in('/ios/')"
    agent:
      machine:
        type: a1-standard-4
        os_image: macos-mojave-xcode11
    task:
      jobs:
        - commands:
            - checkout
            - cd ios
            - make test

It should be:

  - name: Test iOS client
    dependencies: []
    run:
      when: "change_in('/ios/')"
    task:
      agent:
        machine:
          type: a1-standard-4
          os_image: macos-mojave-xcode11
      jobs:
        - commands:
            - checkout
            - cd ios
            - make test

@shiroyasha
Copy link

shiroyasha commented May 10, 2020 •
edited
Loading

I saw that I can enable the agent per task, but that's not really ideal since the iOS block will have more tasks added (building and shipping).

A block can have only one task, but a task can have multiple jobs. When you define an agent on the task level, it will define the agent for all jobs.

@LaloLoop thanks for this input. If I understand correctly, you would like to have multiple sequential tasks on the block level?

@LaloLoop
Copy link

Thank you, @shiroyasha for the quick reply. I ended up configuring it as in the second snippet you provided.
Regarding the agent section, I think it is ok to be able to add it in a per task basis, I was misunderstanding the configuration required, and thought the block would store all the jobs related to the iOS App, but instead I need to create more blocks and link them through dependencies, e.g. An hypothetical deployment block should depend on the tests block.
I am quite new to semaphoreci, so I guess I will try my setup with the provided fix and will let you know if the use case requires something else. Thanks!

@LaloLoop
Copy link

Now that I have been using this configuration more, I came up with the following issue. The

run:
  when: "change_in('/ios/')"

part, is not working, the builds are always executed for all the defined blocks in the configuration, I was wondering if I am specifying the path wrong or if I need to quote the value for when, this is my current configuration.
You can see example runs below, where both the python project and iOS project are always ran, despite the changes being only on the iOS folder.

Thank you!

@shiroyasha
Copy link

shiroyasha commented May 13, 2020 •
edited
Loading

@LaloLoop I've checked it with the rest of the team. By default, when a Semaphore Yaml file is changed in the branches/PRs history, every block is rerun.

We introduced this because changes in the YAML file can significantly impact the behavior of the pipeline.

To ignore changes in the YAML file, you can use:

change_in('/ios', {pipeline_file: 'ignore'})

Thanks for bringing up this issue! Based on your input we have opened a conversation topic about this behavior. We can either be more explicit why was the block executed, or change the defaults of for the change_in syntax.

@katoquro
Copy link

Hello!
I just want to say that if information about behavior described by @shiroyasha would be placed at the top of the guide it could save me an hour ))))

@lucasklaassen
Copy link

Also adding in here that the information in this comment: #1 (comment) should be surfaced within the semaphoreci documentation as it will save engineers countless hours of debugging.

@CurryFishBalls9527
Copy link

Is the change_in('/ios', {pipeline_file: 'ignore'}) format correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@markoa @schmijos @steffenmllr @jdinard @katoquro @shiroyasha @CurryFishBalls9527 @LaloLoop @lucasklaassen @philippks