Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restarting replicated after using the studio results in endless creation of piper containers #4

Open
naderman opened this issue Mar 12, 2017 · 2 comments
Open

Restarting replicated after using the studio results in endless creation of piper containers #4

naderman opened this issue Mar 12, 2017 · 2 comments

Comments

@naderman
Copy link

This starts if I restart the replicated service on the host, or e.g. reboot the machine, and doesn't stop.

45512590407e41415fdacc66b8fb   0 B (virtual 127.8 MB)
fcd9defc57f2        f08b0774069e                                     "entrypoint.sh piper "   20 minutes ago       Exited (255) 20 minutes ago                                                           piper-b7855307564140966f884c95b8034790   0 B (virtual 127.8 MB)
c45e7045012a        f08b0774069e                                     "entrypoint.sh piper "   20 minutes ago       Exited (255) 20 minutes ago                                                           piper-91ccb35e545841db5b6f5be7c28a6130   0 B (virtual 127.8 MB)
6999c92834ad        f08b0774069e                                     "entrypoint.sh piper "   20 minutes ago       Exited (255) 20 minutes ago                                                           piper-81e1e92cd41d495d5639944d46d5f695   0 B (virtual 127.8 MB)
6d0840c29577        f08b0774069e                                     "entrypoint.sh piper "   20 minutes ago       Exited (255) 20 minutes ago                                                           piper-4f91d4a78af146bf77d05bae2402f950   0 B (virtual 127.8 MB)
c3193643526a        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 20 minutes ago                                                           piper-7d64e58dceeb41635456110f289218c8   0 B (virtual 127.8 MB)
8bd035d25ff9        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-8d6751fb8a8043f77c7f505ab0552a4b   0 B (virtual 127.8 MB)
82bd806e8a12        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-e390ad746aaf41ea7f2a9ccee9be2569   0 B (virtual 127.8 MB)
d8671164d257        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-54a32b80bc3c411572052b6761857970   0 B (virtual 127.8 MB)
b905c9e905de        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-e503dede69164e8871647da64d61cf80   0 B (virtual 127.8 MB)
d4bdc9ccf598        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-8102ac3fa490407c75701c57315a505e   0 B (virtual 127.8 MB)
7a734b5eff74        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-17c3bea5278f4fe174557fcfc97a7ea9   0 B (virtual 127.8 MB)
f7d30235cf25        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-4e65c7e17f6246b547eb2def3a9a395c   0 B (virtual 127.8 MB)
d65de917bb0e        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-8067e4dd5a0a41ac77729bbb27562005   0 B (virtual 127.8 MB)
2ed24b71528b        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-33aa25fe650b4b085fecb48126902e45   0 B (virtual 127.8 MB)
47549816ede1        f08b0774069e                                     "entrypoint.sh piper "   21 minutes ago       Exited (255) 21 minutes ago                                                           piper-3aa0f9f952e245d26b27016393413578   0 B (virtual 127.8 MB)
dfdaf8197c78        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 21 minutes ago                                                           piper-0ebe5e7cab9a4f5875e424bd7f56d9f2   0 B (virtual 127.8 MB)
ad2c25803a42        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-a5d4ddcf0d6b4687450597bdeb49acaf   0 B (virtual 127.8 MB)
3af9546c963d        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-b4a56122535445f6632f4473b234265d   0 B (virtual 127.8 MB)
cf6656ecd71d        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-a4b33b54dfd246824efd11dc343b41e1   0 B (virtual 127.8 MB)
7bcd37bdb8c7        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-9ed529bbf6ac44fe5e6b3dcb05bc90bb   0 B (virtual 127.8 MB)
9e87b6e9601c        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-580000db9001430d57af04f384f21c3f   0 B (virtual 127.8 MB)
1144237a2407        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-946b6effc142419b4a013cffef9d6cfc   0 B (virtual 127.8 MB)
01c849b1258d        f08b0774069e                                     "entrypoint.sh piper "   22 minutes ago       Exited (255) 22 minutes ago                                                           piper-559a

Syslog:

Mar 12 20:39:31 ubuntu-xenial docker[2805]: ERROR 2017/03/12 20:39:31 premkit/log/gin.go:52 [GIN] 500 | 263.971221ms |  | GET     /v0.1/scheduler/snapshots/cluster-node-restore/stream                            
Mar 12 20:39:31 ubuntu-xenial docker[2771]: ERROR 2017/03/12 20:39:31 eventsource/eventsource.go:54 Eventsource stream https://replicated/v0.1/scheduler/snapshots/cluster-node-restore/stream?appid=[app id]&version=42 subscription error: 500:                                                                                                                                                             
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.621553] docker0: port 2(veth26fcf92) entered disabled state                                                                                                           
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.621614] veth809a8c1: renamed from eth0                                                                                                                                
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.647837] docker0: port 2(veth26fcf92) entered disabled state                                                                                                           
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.652369] device veth26fcf92 left promiscuous mode                                                                                                                      
Mar 12 20:39:31 ubuntu-xenial kernel: [ 1959.652374] docker0: port 2(veth26fcf92) entered disabled state                                                                                                           
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.492449] aufs au_opts_verify:1597:dockerd[6312]: dirperm1 breaks the protection by the permission bits on the lower branch                                             
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.515465] aufs au_opts_verify:1597:dockerd[6312]: dirperm1 breaks the protection by the permission bits on the lower branch                                             
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.536889] aufs au_opts_verify:1597:dockerd[5506]: dirperm1 breaks the protection by the permission bits on the lower branch                                             
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.546442] device vethe499e34 entered promiscuous mode                                                                                                                   
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.546494] IPv6: ADDRCONF(NETDEV_UP): vethe499e34: link is not ready                                                                                                     
Mar 12 20:39:33 ubuntu-xenial systemd-udevd[4851]: Could not generate persistent MAC address for veth9ed8cb0: No such file or directory                                                                            
Mar 12 20:39:33 ubuntu-xenial systemd-udevd[4852]: Could not generate persistent MAC address for vethe499e34: No such file or directory                                                                            
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.649525] eth0: renamed from veth9ed8cb0                                                                                                                                
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.660773] IPv6: ADDRCONF(NETDEV_CHANGE): vethe499e34: link becomes ready                                                                                                
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.660794] docker0: port 2(vethe499e34) entered forwarding state                                                                                                         
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.660802] docker0: port 2(vethe499e34) entered forwarding state                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]: INFO 2017-03-12T20:39:33+00:00 [pkg/piper] cmd.go:164 Piper result failed for file                                                                                     
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .dashboard .btn-warning {                                                                                                                                              
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #f0ad4e;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border-color: #eea236;                                                                                                                                               
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #fff;                                                                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .dashboard .btn-warning:hover {                                                                                                                                        
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color:#fff;                                                                                                                                                          
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color:#ec971f;                                                                                                                                            
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border-color:#d58512;                                                                                                                                                
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .react-grid-item {                                                                                                                                                     
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border: 0;                                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* cpu graphs */                                                                                                                                                       
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-cpuacct {                                                                                                                                              
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #ec663c;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-cpuacct .widget-metrics-chart {                                                                                                                        
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #ec663c;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-cpu-metrics > .text-muted {                                                                                                                                    
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #ccc;                                                                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* memory graphs */                                                                                                                                                    
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-memory {                                                                                                                                               
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #8A4F7D;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-memory .widget-metrics-chart {                                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #8A4F7D;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-memory-metrics > .text-muted {                                                                                                                                 
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #ccc;                                                                                                                                                         
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App status tile */                                                                                                                                                  
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appstatus {                                                                                                                                                    
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #2A747B;                                                                                                                                           
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }                                                                                                                                                                      
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App udpate tile */                                                                                                                                                  
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appupdate {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #12B0C5;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App backup tile */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appbackup {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #887880;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /manifest.json: lstat /host/
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .dashboard .btn-warning {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #f0ad4e;
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border-color: #eea236;
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #fff;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .dashboard .btn-warning:hover {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color:#fff;
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color:#ec971f;
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border-color:#d58512;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .react-grid-item {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   border: 0;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* cpu graphs */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-cpuacct {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #ec663c;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-cpuacct .widget-metrics-chart {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #ec663c;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-cpu-metrics > .text-muted {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #ccc;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* memory graphs */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-memory {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #8A4F7D;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-metrics-memory .widget-metrics-chart {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #8A4F7D;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-memory-metrics > .text-muted {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   color: #ccc;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App status tile */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appstatus {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #2A747B;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App udpate tile */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appupdate {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #12B0C5;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /* App backup tile */
Mar 12 20:39:33 ubuntu-xenial docker[2805]: .widget-appbackup {
Mar 12 20:39:33 ubuntu-xenial docker[2805]:   background-color: #887880;
Mar 12 20:39:33 ubuntu-xenial docker[2805]: }
Mar 12 20:39:33 ubuntu-xenial docker[2805]: /manifest.json: no such file or directory
Mar 12 20:39:33 ubuntu-xenial docker[2805]: ERROR 2017/03/12 20:39:33 premkit/log/gin.go:52 [GIN] 500 | 262.991283ms |  | GET     /v0.1/scheduler/snapshots/cluster-node-restore/stream
Mar 12 20:39:33 ubuntu-xenial docker[2771]: ERROR 2017/03/12 20:39:33 eventsource/eventsource.go:54 Eventsource stream https://replicated/v0.1/scheduler/snapshots/cluster-node-restore/stream?appid=[app id][&version=42 subscription error: 500:
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.769419] docker0: port 2(vethe499e34) entered disabled state
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.769505] veth9ed8cb0: renamed from eth0
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.793862] docker0: port 2(vethe499e34) entered disabled state
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.796901] device vethe499e34 left promiscuous mode
Mar 12 20:39:33 ubuntu-xenial kernel: [ 1961.796905] docker0: port 2(vethe499e34) entered disabled state

This, too loops endlessly
@ferozeb
Copy link

ferozeb commented Dec 14, 2017

I too am seeing this issue.

@emosbaugh
Copy link
Member

hi @ferozeb @naderman are you still experiencing this issue? would it be possible to run docker inspect and docker logs for any of those piper instances that have failed? thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@naderman @emosbaugh @ferozeb