Known Issues and Work Arounds in Cloudera Manager 4
The following sections describe the current known issues and fixed issues in each Cloudera Manager release.
Known Issues in the Current Release
— Replication between clusters in different Kerberos Realms may fail after upgrade.
Upon upgrade to CM 4.6 (from CM 4.5, 4.5.1, or 4.5.2), any existing and new replications between clusters in different Kerberos realms may start failing. Due to a Java 6 bug with cross-realm authentication, the underlying functionality for replication between clusters in different Kerberos realms has changed in Cloudera manager 4.6. Upon upgrade to CM 4.6, you should follow the steps at ../Cloudera-Backup-Disaster-Recovery/cmbdr-replication-and-kerberos.html#xd_583c10bfdbd326ba-5676e95c-13ed333c3d9--7ff3 to set up replication between secure clusters in different realms.
Severity: Med
Anticipated Resolution: To be fixed in an upcoming release.
Workaround: Follow the stepinstructions at ../Cloudera-Backup-Disaster-Recovery/cmbdr-replication-and-kerberos.html#xd_583c10bfdbd326ba-5676e95c-13ed333c3d9--7ff3 to set up replication between clusters in different kerberos realms.
— Setting up Federation causes the JournalNode to be reformatted.
When setting up Federation, the new NameNodes should be configured (via the safety valve) with a QuorumJournal URL that has a different journal name from the original namespace. If the journal name is not changed, the JournalNode directory will be reformatted.
Severity: High
Anticipated Resolution: To be fixed in an upcoming release.
Workaround: Ensure that the QuorumJournal for the new Nameservice has a different name than the original QuorumJournal.
— WebHCat role logs cannot be written in CDH4.2.
When using WebHCat with default configuration on CDH4.2, role logs cannot be written due to permission error on /var/log/hcatalog because it is owned by user hcatalog, not user hive.
Severity: Low
Resolution: Fixed in CDH4.3, where /var/log/hcatalog is owned by the hive user by default.
Workaround: chown /var/log/hcatalog to the process user used by Hive Service, which is hive by default. Alternatively, change the webhcat log directory.
— Impala Queries fail when "Bypass Hive Metastore Server" option is selected.
Impala queries fail on CDH4.1 when Hive "Bypass Hive Metastore Server" option is selected. You can work around this by using the Impala Safety Valve for hive-site.xml, replacing <hive_metastore_server_host> with the name of your Hive metastore server host.
Severity: Med
Anticipated Resolution: Fixed in CDH4.2. No plans to fix for CDH4.1.
Workaround: See the detailed instructions for the safety valve configuration in Installing Impala with Cloudera Manager.
— Hive Table Stats configuration recommended for optimal performance.
Configuring Hive Table Stats is highly recommended when using Impala. It allows Impala to make optimizations that can result in significant (over 10x) performance improvements for some joins. If these are not available, Impala will still function, but at lower performance.
Severity: Med
Anticipated Resolution: To be fixed in an upcoming release.
Workaround: See Installing Impala with Cloudera Manager in the Cloudera Manager Installation Guide for information on configuring Hive Table Stats.
— Health Check for Navigator and Reports appears in the API results even if those roles are not configured.
The Cloudera Manager Navigator health check appears as "Not Available" in the Cloudera Manager API health results for the MGMT service, even if no Navigator role is configured. The same is true of the Reports Manager role. This can occur if you are running the Cloudera Standard version of Cloudera Manager. This can be safely ignored and may be removed in a future release.
Severity: Medium
Anticipated Resolution: To be fixed in an upcoming release.
Workaround: None.
— During HDFS replication, tasks may fail due to DataNode timeouts.
In CDH4.2, during an HDFS replication job (using Cloudera Manager's Backup and Data Recovery product) individual tasks in the Replication job may fail due to DataNode timeouts. If enough of these timeouts occur, the replication task may slow down, and the entire replication job could time out and fail.
Severity: Medium
Anticipated Resolution: To be fixed in an upcoming CDH release.
Workaround: None.
— Upgrading a secure CDH3 cluster to CDH4 fails because of missing HTTP principal in NameNode's keytab
If you have set up a secure CDH3 cluster using a Cloudera Manager version before 4.5, upgrading the cluster to CDH4 will fail because the NameNode's hdfs.keytab file does not contain the HTTP principal that is required in CDH4 HDFS.
If using a custom keytab generating script with Cloudera Manager, the script should be modified to include the HTTP principal for CDH3 NameNodes to enable an upgrade to CDH4.
Severity: High if you used a pre-4.5 CM to set up a secure CDH3 cluster and want to upgrade it to CDH4. Otherwise N/A.
Workaround:
- Upgrade to Cloudera Manager 4.5 or later.
- From the Administration menu, select Kerberos.
- Select the NameNode's credentials and press the Regenerate button. This will cause the HTTP principal to be included in the NameNode's hdfs.keytab.
Note that if you set up a secure CDH3 cluster using Cloudera Manager 4.5, this workaround is not necessary and the bug does not manifest.
— NameNode cannot use wildcard address in a secure cluster
In a secure cluster, you cannot use a wildcard for the NameNode's RPC or HTTP bind address. For example, dfs.namenode.http-address must be a real, routable address and port, not 0.0.0.0.<port>. This should affect you only if you are running a secure cluster and your NameNode needs to bind to multiple local addresses.
Bug: HDFS-4448
Severity: Medium
Anticipated Resolution: To be fixed in an upcoming release.
Workaround: None.
— Java 6 GC bug leads to a memory leak.
Java 6 has a bug with finalizers that leads to a memory leak when using -XX:+ConcMarkSweepGC. This bug is fixed in Java6u32. See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7112034. To work around this JVM bug, Cloudera Manager configures processes with both -XX:+ConcMarkSweepGC and -XX:-CMSConcurrentMTEnabled. This workaround has a slight performance penalty.
Severity: Med
Anticipated Resolution: None in Cloudera Manager; fixed in Java6u32.
Workaround: As described above. If you have a JVM that does not exhibit this bug, you can remove -XX:-CMSConcurrentMTEnabled by configuring the JVM arguments for your services.
— After upgrading from 4.5 beta to 4.5 GA, the Hive service lacks a mandatory dependency on a MapReduce service.
After upgrading from 4.5 beta to 4.5 GA, the Hive service lacks a mandatory dependency on a MapReduce service.
Severity: Low
Anticipated Resolution: None.
Workaround: Navigate to the Hive service's configuration page (and dismiss any error popups in the meantime), and set the "MapReduce Service" configuration.
— When upgrading to Cloudera Manager 4.5 the new Hive Metastore Server will fail to start if Hue/Beeswax uses a Derby metastore
When upgrading to Cloudera Manager 4.5, it creates a new Hive service to capture the Hive dependency of an existing Hue service. If Hue/Beeswax uses a Derby metastore, Hue will keep working, but the new Hive Metastore Server will fail to start because a Derby metastore cannot be shared between multiple services. This is harmless. But you should consider migrating away from a Derby metastore.
Severity: Low
Anticipated Resolution: None. This is the intended behaviour.
Workaround: None.
— On a Cloudera Manager managed cluster, the NameNode doesn't listen on loopback by default
By default, wildcards are disabled. To have a role (for example, NameNode) listen on loopback, enable wildcards for that role.
Severity: Low
Anticipated Resolution: None.
Workaround: To have a role listen on loopback (for example NameNode) enable wildcards for that role. This can be done from the Configuration tab for all HDFS roles and for the JobTracker.
— If HDFS NameNode is configured to bind to wildcard address using configuration "Bind NameNode to Wildcard Address", certain Hue applications will not work.
The Oozie, JobDesigner, FileBrowser, and Pig Shell applications fail if the NameNode is configured to bind to wildcard address.
Severity: Low
Anticipated Resolution: To be fixed in a future CDH release.
Workaround: Disable NameNode's configuration to bind to wildcard address.
— Installing on AWS, you must use private EC2 hostnames
When installing on an AWS instance, and adding hosts using their public names, the installation will fail when the hosts fail to heartbeat.
Severity: Med
Anticipated Resolution: To be fixed in a future release.
Workaround:
Use the Back button in the wizard to return to the original screen, where it prompts for a license.
Rerun the wizard, but choose "Use existing hosts" instead of searching for hosts. Now those hosts show up with their internal EC2 names.
Continue through the wizard and the installation should succeed.
— After removing and then re-adding a service, the alternatives settings are incorrect.
After deleting a Cloudera Manager service, the alternatives settings are not cleaned up. If you then re-add the service, it will be given a new instance name, and a new set of configurations settings are added. However, because both the new and old (deleted) instances have the same alternatives priority, the original one will be used rather than the newer one.
Severity: Med
Anticipated Resolution: To be fixed in a future release.
Workaround: The simplest way to fix this is:
- Go to the Configuration tab for the new service instance in Cloudera Manager
- Search for "alternatives"
- Raise the priority value and Save your setting.
- Redeploy your client configuration (from the Actions menu).
— New schema extensions have been introduced for Oozie in CDH4.1
In CDH4.1, Oozie introduced new versions for Hive, Sqoop and workflow schemas. To use them, you must add the new schema extensions to the Oozie SchemaService Workflow Extension Schemas configuration property in Cloudera Manager.
Severity: Low
Anticipated Resolution: To be fixed in a future release.
Workaround: In Cloudera Manager, do the following:
- Go to the CDH4 Oozie service page.
- Go to the Configuration tab.
- Select the Oozie Server category.
- Add the following to the Oozie SchemaService Workflow Extension
Schemas
property:
shell-action-0.2.xsd hive-action-0.3.xsd sqoop-action-0.3.xsd
- Save these changes.
— In Add Services, Impala appears as an option even if the packages have not been installed.
In the Add Services wizard, Impala will appear as a choice even if the Impala packages have not been installed. This can occur if you have installed CDH4.0 rather than CDH4.1, or if you are running an operating system that Impala does not support (such as Ubuntu or SLES), or if you have simply not installed the Impala packages. The Add Service process will appear to succeed, but the service will not start, and eventually times out with an error.
Severity: Low
Anticipated Resolution: To be fixed in a future release.
Workaround: None.
— Stop dependent HBase services before enabling HDFS Automatic Failover.
When enabling HDFS Automatic Failover, you need to first stop any dependent HBase services. The Automatic Failover configuration workflow restarts both NameNodes, which could cause HBase to become unavailable.
Severity: Medium
Anticipated Resolution: To be fixed in a future release.
— On Ubuntu 10.04, the Cloudera Manager agent will not run with an upgraded system python.
On Ubuntu 10.04, the Cloudera Manager agent will not run if the system python is upgraded to 2.6.5-1ubuntu6.1. (2.6.5-1ubuntu6 works correctly.) If you have upgraded, you must also rebuild your pre-prepared virtualenv.
Severity: Medium
Anticipated Resolution: None.
Workaround: Run the following commands:
# apt-get install python-virtualenv # virtualenv /usr/lib64/cmf/agent/build/env
— Cloudera Manager does not support encrypted shuffle.
Encrypted shuffle has been introduced in CDH4.1, but it is not currently possible to enable it through Cloudera Manager.
Severity: Medium
Anticipated Resolution: To be fixed in a future release.
Workaround: None.
— Enabling or disabling High Availability requires Hive Metastore modifications.
Enabling or disabling High Availability for HDFS NameNode requires the Hive Metastore to be modified. This is necessary if the cluster consists of services that depend on Hive, such as Impala and Hue. To modify the Hive Metastore before proceeding with Enabling or Disabling HDFS High Availability, see the Known Issue "Tables created in Hive/Beeswax before HDFS is converted to HA become inaccessible after a failover" in the CDH4 Release Notes for more information.
Severity: Medium
Anticipated Resolution: To be fixed in a future release.
Workaround: Run the "Update Hive Metastore NameNodes" command under the Hive service.
— Impala cannot be used with Federated HDFS
If your cluster is configured to use Federated HDFS, Impala queries will fail.
Severity: Low
Anticipated Resolution: To be fixed in a future release.
Workaround: None.
— Links from the HBase Master Web UI to RegionServer Web UIs may be incorrect.
In order for the links from the HBase Master Web UI to the RegionServer Web UIs to be correct, all the RegionServer Web UI ports must be the same. These can be different from default value of 60030, but all must use the same port number. For the RegionServer Web UI port configuration, roletype and role level values should all be the same.
Severity: Low
Anticipated Resolution: To be fixed in a future release.
Workaround: Links from Cloudera Manager to the RegionServer Web UIs will be correct, and can be used to access the RegionServer Web UIs if the web ports cannot be the same.
— If HDFS uses Quorum-based Storage but does not have HA enabled, the SecondaryNameNode will be unable to checkpoint.
If HDFS is set up in non-HA mode, but with Quorum-based storage configured, the dfs.namenode.edits.dir is automatically configured to the Quorum-based Storage URI. However, the SecondaryNameNode cannot currently read the edits from a Quorum-based Storage URI, and will be unable to do a checkpoint.
Severity: Medium
Anticipated Resolution: To be fixed in a future release
Workaround: Add to the NameNode's safety valve the dfs.namenode.edits.dir property with both the value of the Quorum-based Storage URI as well as a local directory, and restart the NameNode. For example,
<property> <name>dfs.namenode.edits.dir</name> <value>qjournal://jn1HostName:8485;jn2HostName:8485;jn3HostName:8485/journalhdfs1,file:///dfs/edits</value> </property>
— Changing the rack configuration may temporarily cause mis-replicated blocks to be reported.
A rack re-configuration will cause HDFS to report mis-replicated blocks until HDFS rebalances the system, which may take some time. This is a normal side-effect of changing the configuration.
Severity: Low
Anticipated Resolution: None
Workaround: None
— "Use Embedded Database" option in upgrade wizard doesn't display server host names.
When upgrading Cloudera Manager Free Edition 4.0 to Cloudera Manager Enterprise Edition 4.0 on Ubuntu 10.04, the Database Setup page in the upgrade wizard will not display the server host names when you use the "Use Embedded Database" option.
Severity: Low
Anticipated Resolution: To be fixed in a future release
Workaround: Switch to "Custom" and type the host name. It will be the same host name as the Cloudera Manager Server's host name.
— While starting an HDFS service with High Availability and Automatic Failover enabled, one of the NameNodes might not start up.
Severity: Low
Anticipated Resolution: To be fixed in a future release
Workaround: To fix this, start the NameNode that failed to start up after the remaining HDFS roles start up.
— The Federated HDFS Nameservice feature doesn't support nested mount points, so '/' cannot be used as a mount point.
A Federated HDFS Service doesn't support nested mount points, so it is impossible to mount anything at '/'. Note that because of this issue, the root directory will always be read-only, and any client application that requires a writeable root directory will fail.
Severity: Low
Anticipated Resolution: To be fixed in a future release
- In the CDH4 HDFS Service > Configuration tab of the Cloudera Manager Admin Console, search for "nameservice".
- In the Mountpoints field, change the mount point from "/" to a list of mount points that are in the namespace that the Nameservice will manage. (You can enter this as a comma-separated list - for example, "/hbase, /tmp, /user" or by clicking the plus icon to add each mount point in its own field.) You can determine the list of mount points by running the command hadoop fs -ls / from the CLI on the NameNode host.
— In the HDFS service, the default value for the Superuser Group setting has changed.
The default value for the Superuser Group setting (dfs.permissions.supergroup and dfs.permissions.superusergroup) has changed. In Cloudera Manager 3.7, the default value was hadoop. In Cloudera Manager 4.0, the default value is now superuser.
Severity: Low
Anticipated Resolution: None
Workaround: If necessary, you can change the value for the Superuser Group by setting it in the HDFS > Configuration tab of the Cloudera Manager Admin Console.
— After upgrading to CM 4.1, roles may need to be restarted for Log Directory Monitoring to work.
After upgrading to Cloudera Manager 4.1, directory monitoring may show status "UNKNOWN" until roles are restarted. You can either restart the roles, or just ignore the unknown status until the next planned restart.
Severity: Low
Anticipated Resolution: To be fixed in a future release.
Workaround: None.
— Historical disk usage reports do not work with federated HDFS.
Severity: Low
Anticipated Resolution: To be fixed in a future release.
Workaround: None.
— (Applies to CDH4 only) Activity monitoring does not work on YARN activities.
Severity: Low
Anticipated Resolution: To be fixed in a future release
Workaround: None
— (Applies to CDH3 only) Uninstalling Oozie components in the wrong order will cause the uninstall to fail.
If you uninstall hue-oozie-auth-plugin (which was originally installed with Cloudera Manager 3.7) after uninstalling Oozie, the uninstall hue-oozie-auth-plugin operation will fail and the hue-oozie-auth-plugin package will not be uninstalled.
Severity: Low
Anticipated Resolution: None Work-around: Uninstall hue-oozie-auth-plugin before uninstalling Oozie. If you already attempted to uninstall hue-oozie-auth-plugin after Oozie, you must reinstall Oozie, uninstall hue-oozie-auth-plugin, and then uninstall Oozie again.
— HDFS monitoring configuration applies to all nameservices
The monitoring configurations at the HDFS level apply to all nameservices. So, if there are two federated nameservices, it's not possible to disable a check on one but not the other. Likewise, it's not possible to have different thresholds for the two nameservices.
Severity: Low
Anticipated Resolution: To be fixed in a future release
Workaround: None
— Task details don't appear for CDH4 MR jobs in the Activity Monitor.
In the Activity Monitor, clicking on the Task details for a job sometimes returns "No results found. Try expanding the time range". This is because there is a time lag between when the Activity information appears and its Task details are available.
Severity: Low
Anticipated Resolution: To be fixed in a future release.
Workaround: Wait for bit and try again – results can take up to a full minute to appear.
— Hue's Authman support is not deprecated properly.
For users of Hue with custom configurations for Authorization Manager, after upgrade to Cloudera Manager 4 some management roles such as Reports Manager may not start up properly.
Severity: Medium
Anticipated Resolution: To be fixed in a future release
Workaround: Before upgrade, go in Hue's Authorization Manager's Configuration tab, and reset all custom configurations back to the default value. Each configuration should either be empty or say "inherited value".
— In CDH 4.0 and 4.1, for secure clusters only, Hue cannot connect to the Hive Metastore Server.
Severity: Med
Anticipated Resolution: Fixed in CDH 4.2.
Workaround: There are three workarounds:
-
Upgrade to CDH4.2.
-
Use Hue's safety valve for hive-site.xml to configure Hue to directly connect to the Hive Metastore database. These configurations can easily be found by going to the Hive service, selecting a Hive Metastore Server, navigating to the processes page, expanding "show", then clicking on hive-site.xml. You should include the following:
<property> <name>javax.jdo.option.ConnectionURL</name> <value>JDBC_URL</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>DRIVER_NAME</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>HIVE_DB_USER</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>HIVE_DB_PASSWORD</value> </property> <property> <name>hive.metastore.local</name> <value>true</value> </property> <property> <name>datanucleus.autoCreateSchema</name> <value>false</value> </property> <property> <name>datanucleus.metadata.validate</name> <value>false</value> </property> <property> <name>hive.warehouse.subdir.inherit.perms</name> <value>true</value> </property>
-
Select the "Bypass Hive Metastore" option in Hive service configuration, in the Advanced group. This is not the preferred solution because this configures any Hive CLI to bypass the Hive Metastore Server, even though Hive CLI works with Hive Metastore Server.