Troubleshooting Cluster Configuration and Operation

This section contains solutions to some common problems that prevent you from using Cloudera Manager and describes how to use Cloudera Manager notification and log management tools to diagnose problems.

Solutions to Common Problems

Problems

Possible Causes

Solutions

Logs include APPARENT DEADLOCK entries for c3p0.

These deadlock messages are cause by the c3p0 process not making progress at the expected rate. This can indicate either that c3p0 is deadlocked or that its progress is slow enough to trigger these messages. In many cases, progress is occurring and these messages should not be seen as catastrophic.

There are a variety of ways to react to these log entries.

  • You may ignore these messages if system performance is not otherwise affected. Because these entries often occur during slow progress, they may be ignored in some cases.
  • You may modify the timer triggers. If c3p0 is making slow progress, increasing the period of time during which progress is evaluated stop the log entries from occurring. The default time between Timer triggers is 10 seconds and is configurable indirectly by configuring maxAdministrativeTaskTime. For more information, see maxAdministrativeTaskTime.
  • You may increase the number of threads in the c3p0 pool, thereby increasing the resources available to make progress on tasks. For more information, see numHelperThreads.

Starting Services

After you click the Start button to start a service, the Finished status doesn't display.

The host machine is disconnected from the Server, as indicated by missing heartbeats on the Hosts tab.

  • Look at the logs for the service for causes of the problem.
  • Restart the Agents on the hosts where the heartbeats are missing.

After you click Start to start a service, the Finished status displays but there are error messages. The subcommands to start service components (such as JobTracker and one or more TaskTrackers) do not start.

A port specified in the Configuration tab of the service is already being used in your cluster. For example, the JobTracker port is in use by another process.

Enter an available port number in the port property (such as JobTracker port) in the Configuration tab of the service.

There are incorrect directories specified in the Configuration tab of the service (such as the log directory).

Enter correct directories in the Configuration tab of the service.

Logs and Events

For information about problems, check the logs and events:

  • The Cloudera Manager Server and Agent Logs contain information on the server and host agents.
  • Logs present log information for Hadoop services, filtered by role, host, and/or keywords as well log level (severity).
  • The Events tab lets you search for and display events and alerts that have occurred within a selected time range filtered by service, hosts, and/or keywords. See Viewing and Filtering Events.
  • Event and Log search features are also provided for individual user jobs, or for specific service. See Activity Monitoring and Service Monitoring.