shared-object-creation.sh not running despite being able to resolve opensearch container

[mmguero]

Coming from here with @gustavoberman:

Hello there!
I'm reaching this issue coming from idaholab#514
After installing Malcolm from cloned git in an ubuntu 22.04
But the containers referenced in this issue resolve just fine:

$ docker compose exec dashboards-helper curl opensearch:9200
{
  "name" : "opensearch",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "YWEIyADPQEqbcgmCYnRDGA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.15.0",
    "build_type" : "tar",
    "build_hash" : "61dbcd0795c9bfe9b81e5762175414bc38bbcadf",
    "build_date" : "2024-06-20T03:26:49.193630411Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"

$ docker compose exec dashboards curl opensearch:9200
{
  "name" : "opensearch",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "YWEIyADPQEqbcgmCYnRDGA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.15.0",
    "build_type" : "tar",
    "build_hash" : "61dbcd0795c9bfe9b81e5762175414bc38bbcadf",
    "build_date" : "2024-06-20T03:26:49.193630411Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

And all containers seem to be up:

$ ./scripts/status
NAME                           IMAGE                                                COMMAND                  SERVICE              CREATED             STATUS                       PORTS
malcolm-api-1                  ghcr.io/idaholab/malcolm/api:24.06.0                 "/usr/bin/tini -- /u…"   api                  About an hour ago   Up About an hour (healthy)   5000/tcp
malcolm-arkime-1               ghcr.io/idaholab/malcolm/arkime:24.06.0              "/usr/bin/tini -- /u…"   arkime               About an hour ago   Up About an hour (healthy)   8000/tcp, 8005/tcp, 8081/tcp
malcolm-arkime-live-1          ghcr.io/idaholab/malcolm/arkime:24.06.0              "/usr/bin/tini -- /u…"   arkime-live          About an hour ago   Up About an hour             
malcolm-dashboards-1           ghcr.io/idaholab/malcolm/dashboards:24.06.0          "/usr/bin/tini -- /u…"   dashboards           About an hour ago   Up About an hour (healthy)   5601/tcp
malcolm-dashboards-helper-1    ghcr.io/idaholab/malcolm/dashboards-helper:24.06.0   "/sbin/tini -- /usr/…"   dashboards-helper    About an hour ago   Up About an hour (healthy)   28991/tcp
malcolm-file-monitor-1         ghcr.io/idaholab/malcolm/file-monitor:24.06.0        "/usr/bin/tini -- /u…"   file-monitor         About an hour ago   Up About an hour (healthy)   3310/tcp, 8440/tcp
malcolm-filebeat-1             ghcr.io/idaholab/malcolm/filebeat-oss:24.06.0        "/usr/bin/tini -- /u…"   filebeat             About an hour ago   Up About an hour (healthy)   127.0.0.1:5045->5045/tcp
malcolm-freq-1                 ghcr.io/idaholab/malcolm/freq:24.06.0                "/usr/bin/tini -- /u…"   freq                 About an hour ago   Up About an hour (healthy)   10004/tcp
malcolm-htadmin-1              ghcr.io/idaholab/malcolm/htadmin:24.06.0             "/usr/bin/tini -- /u…"   htadmin              About an hour ago   Up About an hour (healthy)   80/tcp
malcolm-logstash-1             ghcr.io/idaholab/malcolm/logstash-oss:24.06.0        "/usr/bin/tini -- /u…"   logstash             About an hour ago   Up About an hour (healthy)   9001/tcp, 127.0.0.1:5044->5044/tcp, 9600/tcp
malcolm-netbox-1               ghcr.io/idaholab/malcolm/netbox:24.06.0              "/usr/bin/tini -- /u…"   netbox               About an hour ago   Up About an hour (healthy)   9001/tcp
malcolm-netbox-postgres-1      ghcr.io/idaholab/malcolm/postgresql:24.06.0          "/sbin/tini -- /usr/…"   netbox-postgres      About an hour ago   Up About an hour (healthy)   5432/tcp
malcolm-netbox-redis-1         ghcr.io/idaholab/malcolm/redis:24.06.0               "/sbin/tini -- /usr/…"   netbox-redis         About an hour ago   Up About an hour (healthy)   6379/tcp
malcolm-netbox-redis-cache-1   ghcr.io/idaholab/malcolm/redis:24.06.0               "/sbin/tini -- /usr/…"   netbox-redis-cache   About an hour ago   Up About an hour (healthy)   6379/tcp
malcolm-nginx-proxy-1          ghcr.io/idaholab/malcolm/nginx-proxy:24.06.0         "/sbin/tini -- /usr/…"   nginx-proxy          About an hour ago   Up About an hour (healthy)   0.0.0.0:443->443/tcp, 127.0.0.1:9200->9200/tcp
malcolm-opensearch-1           ghcr.io/idaholab/malcolm/opensearch:24.06.0          "/usr/bin/tini -- /u…"   opensearch           About an hour ago   Up About an hour (healthy)   9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp
malcolm-pcap-capture-1         ghcr.io/idaholab/malcolm/pcap-capture:24.06.0        "/usr/bin/tini -- /u…"   pcap-capture         About an hour ago   Up About an hour             
malcolm-pcap-monitor-1         ghcr.io/idaholab/malcolm/pcap-monitor:24.06.0        "/usr/bin/tini -- /u…"   pcap-monitor         About an hour ago   Up About an hour (healthy)   30441/tcp
malcolm-suricata-1             ghcr.io/idaholab/malcolm/suricata:24.06.0            "/usr/bin/tini -- /u…"   suricata             About an hour ago   Up About an hour (healthy)   
malcolm-suricata-live-1        ghcr.io/idaholab/malcolm/suricata:24.06.0            "/usr/bin/tini -- /u…"   suricata-live        About an hour ago   Up About an hour             
malcolm-upload-1               ghcr.io/idaholab/malcolm/file-upload:24.06.0         "/usr/bin/tini -- /u…"   upload               About an hour ago   Up About an hour (healthy)   80/tcp, 127.0.0.1:8022->22/tcp
malcolm-zeek-1                 ghcr.io/idaholab/malcolm/zeek:24.06.0                "/usr/bin/tini -- /u…"   zeek                 About an hour ago   Up About an hour (healthy)   
malcolm-zeek-live-1            ghcr.io/idaholab/malcolm/zeek:24.06.0                "/usr/bin/tini -- /u…"   zeek-live            About an hour ago   Up About an hour             

but the log keeps saying:
dashboards-helper-1 | 2024-07-23T17:08:00Z /data/shared-object-creation.sh: not starting: job is still running since 2024-07-23 15:44:00 +0000 UTC (1h24m0s elapsed)

And the dashboard don't have any info, graphs says:
Could not locate that index-pattern-field (id: event.provider)

And have lots of little windows messages with:
No matching indices found: No indices match pattern "arkime_sessions3-*"

Any ideas?
Thanks

[mmguero]

The last two messages you pasted are just a result of the shared-object-creation.sh script not having done its thing yet, and no data being inserted. For what you're seeing we need to figure out why that hasn't happened yet. Let's get the output of ./scripts/logs -s dashboards-helper prior to the "job is still running" stuff. you could also run `docker compose

[mmguero]

Yeah, sometimes I need to restart just a single container when I'm doing development testing on it, and this is the command I use (you use the name of the service in the docker-compose.yml file, so in this example I'm using the api container):

docker compose stop api ; docker compose rm --force api ; docker compose up -d --no-deps api

What I'm not 100% sure of is whether or not it gets updates to the environment variable files. Let me double-check that and get back to you.

[mmguero]

Okay, I just did a quick test using the command I put in my last comment. I edited a value in one of the .env files in my config directory and ran that command to stop, remove, and restart the container. It did get the new environment variable value I added to the .env file, as I hoped it would.

[mmguero]

See idaholab#521 for an enhancement I'm going to put in to handle a service restart with the ./scripts/restart script.

[mmguero]

In the next release, you'll be able to just do ./scripts/restart -s api (or whatever the service you want to restart is) and it should just do that one. Thanks for the thoughts that led to this enhancement, this will actually be useful for me as well.