Creating a Kerberized cluster

• Create a non Kerberized cluster and verify all the relevant components are working. Log in to the Bootstrap node. Confirm the file cookbooks/bcpc-hadoop/attributes/kerberos.rb has [:bcpc][:hadoop][:kerberos][:enable] flag set to false. The rest of the instructions should be run from the Bootstrap node.
• Add the following recipes to the runlist of BCPC-Bootstrap role in roles/BCPC-Bootstrap.json

    "recipe[bach_krb5::krb5_server]",
    "recipe[bach_krb5::keytabs]"

• Update role definitions on chef server

    sudo knife role from file roles/BCPC-Bootstrap.json

• Chef the bootstrap node by running the following chef-client command.

    sudo chef-client -c .chef/knife.rb

• Add the following recipes to BCPC-Hadoop-Worker & BCPC-Hadoop-Head roles insert "recipe[bach_krb5::krb5_client]", before ```"recipe[bcpc-hadoop::configs]",````

• Update role definitions on chef server

    sudo knife role from file roles/BCPC-Hadoop-Worker.json
    sudo knife role from file roles/BCPC-Hadoop-Head.json

• Update cookbooks/bcpc-hadoop/attributes/kerberos.rb to set [:bcpc][:hadoop][:kerberos][:enable] to true and upload the cookbook.

   sudo knife cookbook upload  bcpc-hadoop

• Chef all the nodes by executing cluster-assign-roles.sh. NameNode requires two runs of chef-client.

    ./cluster-assign-roles.sh Test-Laptop hadoop bcpc-vm1 
   ./cluster-assign-roles.sh Test-Laptop hadoop bcpc-vm2 
   ./cluster-assign-roles.sh Test-Laptop hadoop bcpc-vm3
   ./cluster-assign-roles.sh Test-Laptop hadoop bcpc-vm1 

• Add Kerberos principals by using kadmin.local on bootstrap node

    root@bcpc-bootstrap:/home/vagrant/chef-bcpc# kadmin.local
    Authenticating as principal root/[email protected] with password.
    kadmin.local:  add_principal ubuntu
    WARNING: no policy specified for [email protected]; defaulting to no policy
    Enter password for principal "[email protected]":
    Re-enter password for principal "[email protected]":
    Principal "[email protected]" created.
    kadmin.local:

• Log into one of the hadoop cluster nodes and perform kinit for the principal added in the previous step and use the password used while adding the principal

    ubuntu@bcpc-vm1:~$ kinit
    Password for [email protected]:

• Verify that you are able to perform operations on hadoop cluster

    ubuntu@bcpc-vm1:~$ hdfs dfs -ls /
    Found 8 items
    drwxrwxrwt   - yarn   supergroup          0 2015-09-21 20:02 /app-logs
    drwxr-xr-x   - hdfs   supergroup          0 2015-09-21 20:55 /apps
    drwxr-xr-x   - hbase  hadoop              0 2015-09-22 10:52 /hbase
    drwxr-xr-x   - hdfs   hadoop              0 2015-09-21 20:56 /hdp
    drwxr-xr-x   - mapred hdfs                0 2015-09-21 20:31 /mr-history
    drwxrwxrwt   - hdfs   supergroup          0 2015-09-22 10:52 /tmp
    drwxr-xr-x   - hdfs   supergroup          0 2015-09-21 20:40 /user
    drwxr-xr-x   - hdfs   supergroup          0 2015-09-21 20:03 /var

• If the user is not added to Kerberos KDC or if the kinit is not successful for any reason, error similar to the following will be thrown when you try to perform any actions on hadoop services.

    Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

• In the case of HBase, users need to have authorization to perform DDL/DML or admin tasks against HBase objects once the cluster is kerberized. By default only the hbase uses in the VM cluster will be able to perform these tasks. To provide access to perform these tasks to another user, one can do the following assuming the user id which requires all the access is ubuntu.

    $ sudo -u hbase hbase shell
    hbase(main):001:0> grant 'ubuntu', 'RWCA'
     ..skipping standard messages..
     0 row(s) in 0.9490 seconds 

• Another option is to add the user-id or the user group which requires the access to the property hbase.superuser in hbase-site.xml.

  <property>
    <name>hbase.superuser</name>
    <value>hbase,@ubuntu</value>
    <description>List of users or groups, who are allowed full privileges, regardless of stored ACLs, across the cluster. Only used when HBase security is enabled.</description>
  </property>