Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x509 auth always fails unless connection_options: - tlsAllowInvalidHostnames=true #518

Open
davidwartell opened this issue Dec 1, 2022 · 4 comments
Open

x509 auth always fails unless connection_options: - tlsAllowInvalidHostnames=true #518

davidwartell opened this issue Dec 1, 2022 · 4 comments
Labels
bug This issue/PR relates to a bug. help wanted

Comments

@davidwartell
Copy link

davidwartell commented Dec 1, 2022 •
edited
Loading

SUMMARY

x509 auth always fails unless:
connection_options: - tlsAllowInvalidHostnames=true

ISSUE TYPE
  • Bug Report
COMPONENT NAME

x509 auth handling with pymongo

ANSIBLE VERSION

From mysql replica set primary node:

ansible --version
ansible [core 2.13.6]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0]
  jinja version = 3.0.3
  libyaml = True

From controller host:

ansible --version
ansible [core 2.13.6]
  config file = /Users/dwartell/freebird/rwsecure-pod-dev/ansible/ansible.cfg
  configured module search path = ['/Users/dwartell/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/Cellar/ansible/6.6.0/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/dwartell/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible
  python version = 3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)]
  jinja version = 3.1.2
  libyaml = True

Pymongo version on replica set master node

python3 -m pip list | grep mongo
pymongo                4.3.3
COLLECTION VERSION
ansible-galaxy collection list | grep mong
community.mongodb             1.4.2
CONFIGURATION
DEFAULT_HOST_LIST(/Users/dwartell/freebird/rwsecure-pod-dev/ansible/ansible.cfg) = ['/Users/dwartell/freebird/rwsecure-pod-dev/ansible/inventory']
DEFAULT_PRIVATE_KEY_FILE(/Users/dwartell/freebird/rwsecure-pod-dev/ansible/ansible.cfg) = /Users/dwartell/freebird/rwsecure-pod-dev/terraform/certs/bastionhost/id_rsa
DEFAULT_REMOTE_USER(/Users/dwartell/freebird/rwsecure-pod-dev/ansible/ansible.cfg) = ubuntu
OS / ENVIRONMENT

mysql replica set primary node:

cat /proc/version
Linux version 5.15.0-1023-aws (buildd@bos02-arm64-076) (gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #27-Ubuntu SMP Thu Oct 20 16:46:19 UTC 2022
lsb_release -r
Release:	22.04

mongosh --tls --host localhost --tlsCertificateKeyFile /etc/mongo-certs/mongo-rwadmin-user-cert-and-key.pem --tlsCAFile /etc/mongo-certs/mongo-ca-cert.pem --authenticationDatabase '$external' --authenticationMechanism MONGODB-X509 --eval "db.version()"
Current Mongosh Log ID:	6387f46f68b9c93944b9c978
Connecting to:		mongodb://localhost:27017/?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fetc%2Fmongo-certs%2Fmongo-rwadmin-user-cert-and-key.pem&tlsCAFile=%2Fetc%2Fmongo-certs%2Fmongo-ca-cert.pem&authSource=%24external&authMechanism=MONGODB-X509&appName=mongosh+1.6.0
Using MongoDB:		6.0.3
Using Mongosh:		1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2022-11-30T17:58:03.060+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
------

6.0.3

Controller host:
Macos 13.0.1 (22A400) arm64

STEPS TO REPRODUCE
#!/usr/bin/env ansible-playbook

- name: Check replica set status for all replica set nodes are in a healthy state
  hosts: mongo
  gather_facts: no
  tasks:
    - name: check mongo replica set state
      become: true
      community.mongodb.mongodb_status:
        login_host: localhost
        replica_set: rs0
        auth_mechanism: "MONGODB-X509"
        tls: true
        tlsCAFile: /etc/mongo-certs/mongo-ca-cert.pem
        tlsCertificateKeyFile: /etc/mongo-certs/mongo-rwadmin-user-cert-and-key.pem
        login_database: "$external"
        # this tlsAllowInvalidHostnames option is needed or the ansible community mongo ansible collection wont work with x509
        # must be a defect, works correctly with mongosh
        #connection_options:
        #  - "tlsAllowInvalidHostnames=true"
# mongosh --tls --host localhost --tlsCertificateKeyFile /etc/mongo-certs/mongo-rwadmin-user-cert-and-key.pem --tlsCAFile /etc/mongo-certs/mongo-ca-cert.pem --authenticationDatabase '$external' --authenticationMechanism MONGODB-X509 admin
Current Mongosh Log ID:	6387f52f3e68e0f7ec7a7caa
Connecting to:		mongodb://localhost:27017/admin?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fetc%2Fmongo-certs%2Fmongo-rwadmin-user-cert-and-key.pem&tlsCAFile=%2Fetc%2Fmongo-certs%2Fmongo-ca-cert.pem&authSource=%24external&authMechanism=MONGODB-X509&appName=mongosh+1.6.0
Using MongoDB:		6.0.3
Using Mongosh:		1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2022-11-30T17:58:03.060+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
------

rs0 [direct: primary] admin> db.getSiblingDB("$external").auth({mechanism: "MONGODB-X509", user: "[email protected],CN=rwadmin,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US"})
{ ok: 1 }
rs0 [direct: primary] admin> db.getSiblingDB("$external").getUsers();
{
  users: [
    {
      _id: '[email protected],CN=rwadmin,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US',
      userId: new UUID("8599dc7b-8eb6-42c2-82ec-d6babd85c590"),
      user: '[email protected],CN=rwadmin,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US',
      db: '$external',
      roles: [ { role: 'root', db: 'admin' } ],
      mechanisms: [ 'external' ]
    },
    {
      _id: '[email protected],CN=rwapp,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US',
      userId: new UUID("f64f84c2-a6d4-441b-b1d8-7d236851c471"),
      user: '[email protected],CN=rwapp,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US',
      db: '$external',
      roles: [ { role: 'readWrite', db: 'rwsecure' } ],
      mechanisms: [ 'external' ]
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1669854524, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0398d62ed09262dce376cd4aac40801db40a7445", "hex"), 0),
      keyId: Long("7171583554449571845")
    }
  },
  operationTime: Timestamp({ t: 1669854524, i: 1 })
}
rs0 [direct: primary] admin>
EXPECTED RESULTS

It should work like mongosh

# mongosh --tls --host localhost --tlsCertificateKeyFile /etc/mongo-certs/mongo-rwadmin-user-cert-and-key.pem --tlsCAFile /etc/mongo-certs/mongo-ca-cert.pem --authenticationDatabase '$external' --authenticationMechanism MONGODB-X509 admin
Current Mongosh Log ID:	6387f6426814dbd6590f5161
Connecting to:		mongodb://localhost:27017/admin?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=%2Fetc%2Fmongo-certs%2Fmongo-rwadmin-user-cert-and-key.pem&tlsCAFile=%2Fetc%2Fmongo-certs%2Fmongo-ca-cert.pem&authSource=%24external&authMechanism=MONGODB-X509&appName=mongosh+1.6.0
Using MongoDB:		6.0.3
Using Mongosh:		1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2022-11-30T17:58:03.060+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
------

rs0 [direct: primary] admin> db.getSiblingDB("$external").auth({mechanism: "MONGODB-X509", user: "[email protected],CN=rwadmin,OU=mongo-user.dw.dev.stealthsoft.io,O=StealthMode Inc,L=Austin,ST=Texas,C=US"})
{ ok: 1 }
rs0 [direct: primary] admin>

rs0 [direct: primary] admin> rs.status();
{
  set: 'rs0',
  date: ISODate("2022-12-01T00:33:17.870Z"),
  myState: 1,
  term: Long("30"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
    lastCommittedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
    appliedOpTime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
    durableOpTime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
    lastAppliedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
    lastDurableWallTime: ISODate("2022-12-01T00:33:14.626Z")
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1669854784, i: 3 }),
  electionCandidateMetrics: {
    lastElectionReason: 'priorityTakeover',
    lastElectionDate: ISODate("2022-11-30T17:58:24.118Z"),
    electionTerm: Long("30"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1669831103, i: 1 }), t: Long("29") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1669831103, i: 1 }), t: Long("29") },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    priorPrimaryMemberId: 1,
    numCatchUpOps: Long("0"),
    newTermStartDate: ISODate("2022-11-30T17:58:24.224Z"),
    wMajorityWriteAvailabilityDate: ISODate("2022-11-30T17:58:25.243Z")
  },
  members: [
    {
      _id: 0,
      name: 'mgo-wj-a.dw.dev.stealthsoft.io:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 23714,
      optime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDate: ISODate("2022-12-01T00:33:14.000Z"),
      lastAppliedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastDurableWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1669831104, i: 1 }),
      electionDate: ISODate("2022-11-30T17:58:24.000Z"),
      configVersion: 1,
      configTerm: 30,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'mgo-wj-b.dw.dev.stealthsoft.io:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 23713,
      optime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDurable: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDate: ISODate("2022-12-01T00:33:14.000Z"),
      optimeDurableDate: ISODate("2022-12-01T00:33:14.000Z"),
      lastAppliedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastDurableWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastHeartbeat: ISODate("2022-12-01T00:33:16.058Z"),
      lastHeartbeatRecv: ISODate("2022-12-01T00:33:17.097Z"),
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mgo-wj-a.dw.dev.stealthsoft.io:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 30
    },
    {
      _id: 2,
      name: 'mgo-wj-c.dw.dev.stealthsoft.io:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 23713,
      optime: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDurable: { ts: Timestamp({ t: 1669854794, i: 1 }), t: Long("30") },
      optimeDate: ISODate("2022-12-01T00:33:14.000Z"),
      optimeDurableDate: ISODate("2022-12-01T00:33:14.000Z"),
      lastAppliedWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastDurableWallTime: ISODate("2022-12-01T00:33:14.626Z"),
      lastHeartbeat: ISODate("2022-12-01T00:33:16.122Z"),
      lastHeartbeatRecv: ISODate("2022-12-01T00:33:16.116Z"),
      pingMs: Long("1"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mgo-wj-b.dw.dev.stealthsoft.io:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 30
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1669854794, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("a61dcdb51a7939c6f7e972afb4797805c66f5089", "hex"), 0),
      keyId: Long("7171583554449571845")
    }
  },
  operationTime: Timestamp({ t: 1669854794, i: 1 })
}
rs0 [direct: primary] admin>
ACTUAL RESULTS

It fails unless I include these lines in the playbook yaml

        connection_options:
          - "tlsAllowInvalidHostnames=true"

./mongo-status.yml

PLAY [Check replica set status for all replica set nodes are in a healthy state] 

TASK [check mongo replica set state] 
fatal: [mgo-wj-a]: FAILED! => {
  "ansible_facts": {
    "discovered_interpreter_python": "/usr/bin/python3"
  },
  "changed": false,
  "msg": "Unable to determine if auth is enabled: Traceback (most recent call last):\n  File \"/tmp/ansible_community.mongodb.mongodb_status_payload_qsf9hv51/ansible_community.mongodb.mongodb_status_payload.zip/ansible_collections/community/mongodb/plugins/module_utils/mongodb_common.py\", line 279, in is_auth_enabled\n    myclient['admin'].command('listDatabases', 1.0)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/_csot.py\", line 105, in csot_wrapper\n    return func(self, *args, **kwargs)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/database.py\", line 805, in command\n    with self.__client._socket_for_reads(read_preference, session) as (\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py\", line 1296, in _socket_for_reads\n    server = self._select_server(read_preference, session)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/mongo_client.py\", line 1257, in _select_server\n    server = topology.select_server(server_selector)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/topology.py\", line 272, in select_server\n    server = self._select_server(selector, server_selection_timeout, address)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/topology.py\", line 261, in _select_server\n    servers = self.select_servers(selector, server_selection_timeout, address)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/topology.py\", line 223, in select_servers\n    server_descriptions = self._select_servers_loop(selector, server_timeout, address)\n  File \"/usr/local/lib/python3.10/dist-packages/pymongo/topology.py\", line 238, in _select_servers_loop\n    raise ServerSelectionTimeoutError(\npymongo.errors.ServerSelectionTimeoutError: not enough values to unpack (expected 2, got 1), Timeout: 30s, Topology Description: <TopologyDescription id: 6387f1fed957c5ccc26eada3, topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=ValueError('not enough values to unpack (expected 2, got 1)')>]>\n"
}
@rhysmeister rhysmeister mentioned this issue Dec 1, 2022
Closed
@rhysmeister
Copy link
Collaborator

Hello @davidwartell,

Thank you for the very detailed report. Can you also tell us how you generate your certs? Does it work if you use the hostname, i.e. mgo-wj-b.dw.dev.stealthsoft.io?

@fischerscode Have you experienced any issues around this?

Cheers,

Rhys

@rhysmeister rhysmeister added the bug This issue/PR relates to a bug. label Jan 4, 2023
@rhysmeister
Copy link
Collaborator

rhysmeister commented Apr 11, 2023 •
edited
Loading

Local test environment is broken impeding progress on this. Need to fix the versions of kubernetes (and related tools) to the versions in the CI.

@LeoSpyke
Copy link
Contributor

Even if it is not the best solution, I overcame this by adding localhost into the list of SAN(s) in the csr.

To say it in openssl words the outcome is the following:
openssl req -new -newkey rsa:2048 -nodes -keyout replica.key -out replica.csr -addext "extendedKeyUsage = serverAuth, clientAuth" -addext "subjectAltName = DNS:<node1-fqdn>, DNS:<node2-fqdn>, DNS:<node3-fqdn>, DNS:localhost.

Hope it helps.

@rhysmeister
Copy link
Collaborator

This might be an option for a local test env... https://github.com/nektos/act

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue/PR relates to a bug. help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@davidwartell @rhysmeister @LeoSpyke