Using the LZO Parcel
Installing and Activating the Parcel
- Add the appropriate repository to Cloudera Manager’s list of parcel repositories as described in Parcel Configuration Settings. The HADOOP_LZO parcel repositories can be found at: http://archive.cloudera.com/gplextras/parcels/. Install the correct version of the HADOOP_LZO parcel for each Impala version. Starting with Cloudera Manager 4.8, Impala versions 1.2.1 or later are supported; you must use the LZO parcel versions according to the following table:
If required, the repository can be mirrored in the same way as a CDH repo.
Impala Version LZO Parcel Version 1.3.1 HADOOP_LZO-0.4.15-1.gplextras.p0.64 1.2.4 HADOOP_LZO-0.4.15-1.gplextras.p0.57 1.2.3 HADOOP_LZO-0.4.15-1.gplextras.p0.39 1.2.2 HADOOP_LZO-0.4.15-1.gplextras.p0.37 1.2.1 HADOOP_LZO-0.4.15-1.gplextras.p0.33 - Download, distribute, and activate the parcel as described in Managing Parcels.
- Ensure that the lzop.x86_64 and lzo.x86_64 packages are installed.
- Reconfigure and restart services as described in the following sections. Services that do not require the use of LZO need not be reconfigured.
MapReduce
- Go to the MapReduce service.
- Select .
- Search for MapReduce Client Safety.
- In the MapReduce Client Environment Safety Valve, enter the following two lines:
- HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/*
- JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/native
- Search for MapReduce Service Safety.
- In the MapReduce Service
Environment Safety Valve, enter the following two lines:
- HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/*
- JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/native
- Search for io.compression.
- In the Compression Codecs property, click in the field, then click the + sign to open a new value field.
- Add the following two codecs:
- com.hadoop.compression.lzo.LzoCodec
- com.hadoop.compression.lzo.LzopCodec
- Click Save Changes.
- Restart MapReduce.
- Redeploy the MapReduce client configuration.
YARN
- Go to the YARN service.
- Select .
- Search for io.compression.
- In the Compression Codecs property, click in the field, then click the + sign to open a new value field.
- Add the following two codecs:
- com.hadoop.compression.lzo.LzoCodec
- com.hadoop.compression.lzo.LzopCodec
- Expand the category.
- In the YARN Service MapReduce Configuration Safety Valve property, specify:
<property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/*</value> </property> <property> <name>mapreduce.admin.user.env</name> <value>LD_LIBRARY_PATH=/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/native</value> </property>
- Expand the category.
- In the Gateway Client
Environment Safety Valve for hadoop-env.sh, enter the following two
lines:
- HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/*
- JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/native
- Click Save Changes.
- Restart YARN.
- Redeploy the YARN client configuration.
Oozie
- Go to /var/lib/oozie on each host running the Oozie server and symlink the Hadoop LZO JAR /opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar.
Note
: The LZO JAR may already be present in /var/lib/oozie. Replacing any existing JAR with the parcel JAR is strongly recommended. - Restart Oozie.
HBase
Restart HBase.Impala
- On all hosts, install the following packages:
- RHEL 5 / 6 - lzo
- SLES 11, Ubuntu, and Debian - liblzo2-2
- Restart Impala.
Hive
Restart Hive.Sqoop
- Go to the Sqoop service.
- Select .
- Expand the category.
- Add the following entries to the Sqoop Service Environment Safety valve:
- HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/*
- JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/native
- Click Save Changes.
- Restart the Sqoop service.
<< | ||
Terms and Conditions Privacy Policy |