|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.hive.ql.io.HiveFileFormatUtils
public class HiveFileFormatUtils
An util class for various Hive file format tasks. #registerOutputFormatSubstitute(Class, Class) and {@link #getOutputFormatSubstitute(Class)} are added for backward compatibility. They return the newly added HiveOutputFormat for the older ones.
Field Summary | |
---|---|
static String |
READ_COLUMN_IDS_CONF_STR
|
Constructor Summary | |
---|---|
HiveFileFormatUtils()
|
Method Summary | |
---|---|
static boolean |
checkInputFormat(org.apache.hadoop.fs.FileSystem fs,
HiveConf conf,
Class<? extends org.apache.hadoop.mapred.InputFormat> inputFormatCls,
ArrayList<org.apache.hadoop.fs.FileStatus> files)
checks if files are in same format as the given input format |
static Class<? extends InputFormatChecker> |
getInputFormatChecker(Class<?> inputFormat)
get an InputFormatChecker for a file format. |
static org.apache.hadoop.fs.Path |
getOutputFormatFinalPath(org.apache.hadoop.fs.Path parent,
org.apache.hadoop.mapred.JobConf jc,
HiveOutputFormat<?,?> hiveOutputFormat,
boolean isCompressed,
org.apache.hadoop.fs.Path defaultFinalPath)
get the final output path of a given FileOutputFormat. |
static Class<? extends HiveOutputFormat> |
getOutputFormatSubstitute(Class<?> origin)
get a OutputFormat's substitute HiveOutputFormat |
static ArrayList<Integer> |
getReadColumnIDs(org.apache.hadoop.conf.Configuration conf)
Returns an array of column ids(start from zero) which is set in the given parameter conf. |
static void |
registerInputFormatChecker(Class<? extends org.apache.hadoop.mapred.InputFormat> format,
Class<? extends InputFormatChecker> checker)
register an InputFormatChecker for a given InputFormat |
static void |
registerOutputFormatSubstitute(Class<? extends org.apache.hadoop.mapred.OutputFormat> origin,
Class<? extends HiveOutputFormat> substitute)
register a substitute |
static void |
setFullyReadColumns(org.apache.hadoop.conf.Configuration conf)
Clears the read column ids set in the conf, and will read all columns. |
static void |
setReadColumnIDs(org.apache.hadoop.conf.Configuration conf,
ArrayList<Integer> ids)
Sets read columns' ids(start from zero) for RCFile's Reader. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static String READ_COLUMN_IDS_CONF_STR
Constructor Detail |
---|
public HiveFileFormatUtils()
Method Detail |
---|
public static void registerOutputFormatSubstitute(Class<? extends org.apache.hadoop.mapred.OutputFormat> origin, Class<? extends HiveOutputFormat> substitute)
origin
- the class that need to be substitutedsubstitute
- public static Class<? extends HiveOutputFormat> getOutputFormatSubstitute(Class<?> origin)
public static org.apache.hadoop.fs.Path getOutputFormatFinalPath(org.apache.hadoop.fs.Path parent, org.apache.hadoop.mapred.JobConf jc, HiveOutputFormat<?,?> hiveOutputFormat, boolean isCompressed, org.apache.hadoop.fs.Path defaultFinalPath) throws IOException
parent
- parent dir of the expected final output pathjc
- job configuration
IOException
public static void registerInputFormatChecker(Class<? extends org.apache.hadoop.mapred.InputFormat> format, Class<? extends InputFormatChecker> checker)
format
- the class that need to be substitutedchecker
- public static Class<? extends InputFormatChecker> getInputFormatChecker(Class<?> inputFormat)
public static boolean checkInputFormat(org.apache.hadoop.fs.FileSystem fs, HiveConf conf, Class<? extends org.apache.hadoop.mapred.InputFormat> inputFormatCls, ArrayList<org.apache.hadoop.fs.FileStatus> files) throws HiveException
HiveException
public static void setReadColumnIDs(org.apache.hadoop.conf.Configuration conf, ArrayList<Integer> ids)
public static ArrayList<Integer> getReadColumnIDs(org.apache.hadoop.conf.Configuration conf)
public static void setFullyReadColumns(org.apache.hadoop.conf.Configuration conf)
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |