Input: Uses an instance of the specified { [CODESPLIT] public static void setAvroDataSupplier ( Job job , Class < ? extends AvroDataSupplier > supplierClass ) { AvroReadSupport . setAvroDataSupplier ( ContextUtil . getConfiguration ( job ) , supplierClass ) ; }


Input: TODO : making the assumption that getConverter ( i ) is only called once is that valid? [CODESPLIT] @ Override public Converter getConverter ( int fieldIndex ) { // get the real converter from the delegate Converter delegateConverter = checkNotNull ( delegate . getConverter ( fieldIndex ) , "delegate converter" ) ; // determine the indexFieldPath for the converter proxy we're about to make, which is // this converter's path + the requested fieldIndex List < Integer > newIndexFieldPath = new ArrayList < Integer > ( indexFieldPath . size ( ) + 1 ) ; newIndexFieldPath . addAll ( indexFieldPath ) ; newIndexFieldPath . add ( fieldIndex ) ; if ( delegateConverter . isPrimitive ( ) ) { PrimitiveColumnIO columnIO = getColumnIO ( newIndexFieldPath ) ; ColumnPath columnPath = ColumnPath . get ( columnIO . getColumnDescriptor ( ) . getPath ( ) ) ; ValueInspector [ ] valueInspectors = getValueInspectors ( columnPath ) ; return new FilteringPrimitiveConverter ( delegateConverter . asPrimitiveConverter ( ) , valueInspectors ) ; } else { return new FilteringGroupConverter ( delegateConverter . asGroupConverter ( ) , newIndexFieldPath , valueInspectorsByColumn , columnIOsByIndexFieldPath ) ; } }


Input: Calculates the row ranges containing the indexes of the rows might match the specified filter . [CODESPLIT] public static RowRanges calculateRowRanges ( FilterCompat . Filter filter , ColumnIndexStore columnIndexStore , Set < ColumnPath > paths , long rowCount ) { return filter . accept ( new FilterCompat . Visitor < RowRanges > ( ) { @ Override public RowRanges visit ( FilterPredicateCompat filterPredicateCompat ) { try { return filterPredicateCompat . getFilterPredicate ( ) . accept ( new ColumnIndexFilter ( columnIndexStore , paths , rowCount ) ) ; } catch ( MissingOffsetIndexException e ) { LOGGER . info ( e . getMessage ( ) ) ; return RowRanges . createSingle ( rowCount ) ; } } @ Override public RowRanges visit ( UnboundRecordFilterCompat unboundRecordFilterCompat ) { return RowRanges . createSingle ( rowCount ) ; } @ Override public RowRanges visit ( NoOpFilter noOpFilter ) { return RowRanges . createSingle ( rowCount ) ; } } ) ; }


Input: Call this method when setting up your Hadoop job if reading into a Thrift object that is not encoded into the parquet - serialized thrift metadata ( for example writing with Apache Thrift but reading back into Twitter Scrooge version of the same thrift definition or a different but compatible Apache Thrift class ) . [CODESPLIT] public static < T > void setThriftClass ( JobConf conf , Class < T > klass ) { conf . set ( ThriftReadSupport . THRIFT_READ_CLASS_KEY , klass . getName ( ) ) ; }


Input: 1 anonymous element array_element [CODESPLIT] private static GroupType convertArrayType ( final String name , final ListTypeInfo typeInfo ) { final TypeInfo subType = typeInfo . getListElementTypeInfo ( ) ; return listWrapper ( name , listType ( ) , new GroupType ( Repetition . REPEATED , ParquetHiveSerDe . ARRAY . toString ( ) , convertType ( "array_element" , subType ) ) ) ; }


Input: An optional group containing multiple elements [CODESPLIT] private static GroupType convertStructType ( final String name , final StructTypeInfo typeInfo ) { final List < String > columnNames = typeInfo . getAllStructFieldNames ( ) ; final List < TypeInfo > columnTypes = typeInfo . getAllStructFieldTypeInfos ( ) ; return new GroupType ( Repetition . OPTIONAL , name , convertTypes ( columnNames , columnTypes ) ) ; }


Input: 2 elements : key value [CODESPLIT] private static GroupType convertMapType ( final String name , final MapTypeInfo typeInfo ) { final Type keyType = convertType ( ParquetHiveSerDe . MAP_KEY . toString ( ) , typeInfo . getMapKeyTypeInfo ( ) , Repetition . REQUIRED ) ; final Type valueType = convertType ( ParquetHiveSerDe . MAP_VALUE . toString ( ) , typeInfo . getMapValueTypeInfo ( ) ) ; return ConversionPatterns . mapType ( Repetition . OPTIONAL , name , keyType , valueType ) ; }


Input: Expands a string with braces ( {} ) into all of its possible permutations . We call anything inside of {} braces a one - of group . [CODESPLIT] public static List < String > expand ( String globPattern ) { return GlobExpanderImpl . expand ( GlobParser . parse ( globPattern ) ) ; }


Input: / * Creates a new RowRanges object with the single range [ 0 rowCount - 1 ] . [CODESPLIT] static RowRanges createSingle ( long rowCount ) { RowRanges ranges = new RowRanges ( ) ; ranges . add ( new Range ( 0 , rowCount - 1 ) ) ; return ranges ; }


Input: / * Creates a new RowRanges object with the following ranges . [ firstRowIndex [ 0 ] lastRowIndex [ 0 ]] [ firstRowIndex [ 1 ] lastRowIndex [ 1 ]] ... [ firstRowIndex [ n ] lastRowIndex [ n ]] ( See OffsetIndex . getFirstRowIndex and OffsetIndex . getLastRowIndex for details . ) [CODESPLIT] static RowRanges create ( long rowCount , PrimitiveIterator . OfInt pageIndexes , OffsetIndex offsetIndex ) { RowRanges ranges = new RowRanges ( ) ; while ( pageIndexes . hasNext ( ) ) { int pageIndex = pageIndexes . nextInt ( ) ; ranges . add ( new Range ( offsetIndex . getFirstRowIndex ( pageIndex ) , offsetIndex . getLastRowIndex ( pageIndex , rowCount ) ) ) ; } return ranges ; }


Input: / * Calculates the union of the two specified RowRanges object . The union of two range is calculated if there are no elements between them . Otherwise the two disjunct ranges are stored separately . For example : [ 113 241 ] ∪ [ 221 340 ] = [ 113 330 ] [ 113 230 ] ∪ [ 231 340 ] = [ 113 340 ] while [ 113 230 ] ∪ [ 232 340 ] = [ 113 230 ] [ 232 340 ] [CODESPLIT] static RowRanges union ( RowRanges left , RowRanges right ) { RowRanges result = new RowRanges ( ) ; Iterator < Range > it1 = left . ranges . iterator ( ) ; Iterator < Range > it2 = right . ranges . iterator ( ) ; if ( it2 . hasNext ( ) ) { Range range2 = it2 . next ( ) ; while ( it1 . hasNext ( ) ) { Range range1 = it1 . next ( ) ; if ( range1 . isAfter ( range2 ) ) { result . add ( range2 ) ; range2 = range1 ; Iterator < Range > tmp = it1 ; it1 = it2 ; it2 = tmp ; } else { result . add ( range1 ) ; } } result . add ( range2 ) ; } else { it2 = it1 ; } while ( it2 . hasNext ( ) ) { result . add ( it2 . next ( ) ) ; } return result ; }


Input: / * Calculates the intersection of the two specified RowRanges object . Two ranges intersect if they have common elements otherwise the result is empty . For example : [ 113 241 ] ∩ [ 221 340 ] = [ 221 241 ] while [ 113 230 ] ∩ [ 231 340 ] = <EMPTY > [CODESPLIT] static RowRanges intersection ( RowRanges left , RowRanges right ) { RowRanges result = new RowRanges ( ) ; int rightIndex = 0 ; for ( Range l : left . ranges ) { for ( int i = rightIndex , n = right . ranges . size ( ) ; i < n ; ++ i ) { Range r = right . ranges . get ( i ) ; if ( l . isBefore ( r ) ) { break ; } else if ( l . isAfter ( r ) ) { rightIndex = i + 1 ; continue ; } result . add ( Range . intersection ( l , r ) ) ; } } return result ; }


Input: / * Adds a range to the end of the list of ranges . It maintains the disjunct ascending order ( * ) of the ranges by trying to union the specified range to the last ranges in the list . The specified range shall be larger ( * ) than the last one or might be overlapped with some of the last ones . ( * ) [ a b ] < [ c d ] if b < c [CODESPLIT] private void add ( Range range ) { Range rangeToAdd = range ; for ( int i = ranges . size ( ) - 1 ; i >= 0 ; -- i ) { Range last = ranges . get ( i ) ; assert ! last . isAfter ( range ) ; Range u = Range . union ( last , rangeToAdd ) ; if ( u == null ) { break ; } rangeToAdd = u ; ranges . remove ( i ) ; } ranges . add ( rangeToAdd ) ; }


Input: stores a positive long into an int ( assuming it fits ) [CODESPLIT] private int positiveLongToInt ( long value ) { if ( ! ColumnChunkMetaData . positiveLongFitsInAnInt ( value ) ) { throw new IllegalArgumentException ( "value should be positive and fit in an int: " + value ) ; } return ( int ) ( value + Integer . MIN_VALUE ) ; }


Input:
Parse a String into a { @link GlobNodeSequence } [CODESPLIT] public static GlobNodeSequence parse ( String pattern ) { /*
     * The parse algorithm works as follows, assuming we are parsing:
     * "apache{one,pre{x,y}post,two}parquet{a,b}"
     *
     * 1) Begin scanning the string until we find the first {
     *
     * 2) Now that we've found the beginning of a glob group, scan forwards
     *    until the end of this glob group (by counting { and } we see until we find
     *    the closing } for the group we found in step 1).
     *
     * 3) Once the matching closing } is found we need to do two things. First, everything
     *    from the end of the last group up to start of this group is an Atom, so in the example
     *    above, once we've found that "{one,pre{x,y}post,two}" is the first group, we need to grab
     *    "apache" and treat it as an atom and add it to our sequence.
     *    Then, we parse "{one,pre{x,y}post,two}" using a similar but slightly different function (parseOneOf)
     *    and add the result from that to our sequence.
     *
     * 4) Repeat until the end of the string -- so next we find {a,b} and add "parquet" as an Atom and parse
     *    {a,b} using parseOneOf.
     */ if ( pattern . isEmpty ( ) || pattern . equals ( "{}" ) ) { return new GlobNodeSequence ( Arrays . < GlobNode > asList ( new Atom ( "" ) ) ) ; } // the outer parse method needs to parse the pattern into a // GlobNodeSequence, though it may end up being a singleton sequence List < GlobNode > children = new ArrayList < GlobNode > ( ) ; int unmatchedBraces = 0 ; // count of unmatched braces int firstBrace = 0 ; // open brace of current group being processsed int anchor = 0 ; // first un-parsed character position for ( int i = 0 ; i < pattern . length ( ) ; i ++ ) { char c = pattern . charAt ( i ) ; switch ( c ) { case ' ' : if ( unmatchedBraces == 0 ) { // commas not allowed in the top level expression // TODO: maybe turn this check off? throw new GlobParseException ( "Unexpected comma outside of a {} group:\n" + annotateMessage ( pattern , i ) ) ; } break ; case ' ' : if ( unmatchedBraces == 0 ) { // this is the first brace of an outermost {} group firstBrace = i ; } unmatchedBraces ++ ; break ; case ' ' : unmatchedBraces -- ; if ( unmatchedBraces < 0 ) { throw new GlobParseException ( "Unexpected closing }:\n" + annotateMessage ( pattern , i ) ) ; } if ( unmatchedBraces == 0 ) { // grab everything from the end of the last group up to here, // not including the close brace, it is an Atom in our sequence // (assuming it's not empty) if ( anchor != firstBrace ) { // not empty! // (substring's end param is exclusive) children . add ( new Atom ( pattern . substring ( anchor , firstBrace ) ) ) ; } // grab the group, parse it, add it to our sequence, and then continue // note that we skip the braces on both sides (substring's end param is exclusive) children . add ( parseOneOf ( pattern . substring ( firstBrace + 1 , i ) ) ) ; // we have now parsed all the way up to here, the next un-parsed char is i + 1 anchor = i + 1 ; } break ; } } if ( unmatchedBraces > 0 ) { throw new GlobParseException ( "Not enough close braces in: " + pattern ) ; } if ( anchor != pattern . length ( ) ) { // either there were no {} groups, or there were some characters after the // last }, either way whatever is left (could be the entire input) is an Atom // in our sequence children . add ( new Atom ( pattern . substring ( anchor , pattern . length ( ) ) ) ) ; } return new GlobNodeSequence ( children ) ; }


Input: for pretty printing which character had the error [CODESPLIT] private static String annotateMessage ( String message , int pos ) { StringBuilder sb = new StringBuilder ( message ) ; sb . append ( ' ' ) ; for ( int i = 0 ; i < pos ; i ++ ) { sb . append ( ' ' ) ; } sb . append ( ' ' ) ; return sb . toString ( ) ; }


Input: If we are currently writing a bit - packed - run update the bit - packed - header and consider this run to be over [CODESPLIT] private void endPreviousBitPackedRun ( ) { if ( bitPackedRunHeaderPointer == - 1 ) { // we're not currently in a bit-packed-run return ; } // create bit-packed-header, which needs to fit in 1 byte byte bitPackHeader = ( byte ) ( ( bitPackedGroupCount << 1 ) | 1 ) ; // update this byte baos . setByte ( bitPackedRunHeaderPointer , bitPackHeader ) ; // mark that this run is over bitPackedRunHeaderPointer = - 1 ; // reset the number of groups bitPackedGroupCount = 0 ; }


Input: reads one record from in and writes it to out exceptions are not recoverable as record might be halfway written [CODESPLIT] @ Override public void readOne ( TProtocol in , TProtocol out ) throws TException { readOneStruct ( in , out ) ; }


Input: Given a FilterPredicate return a Filter that wraps it . This method also logs the filter being used and rewrites the predicate to not include the not () operator . [CODESPLIT] public static Filter get ( FilterPredicate filterPredicate ) { checkNotNull ( filterPredicate , "filterPredicate" ) ; LOG . info ( "Filtering using predicate: {}" , filterPredicate ) ; // rewrite the predicate to not include the not() operator FilterPredicate collapsedPredicate = LogicalInverseRewriter . rewrite ( filterPredicate ) ; if ( ! filterPredicate . equals ( collapsedPredicate ) ) { LOG . info ( "Predicate has been collapsed to: {}" , collapsedPredicate ) ; } return new FilterPredicateCompat ( collapsedPredicate ) ; }


Input: Given either a FilterPredicate or the class of an UnboundRecordFilter or neither ( but not both ) return a Filter that wraps whichever was provided . <p > Either filterPredicate or unboundRecordFilterClass must be null or an exception is thrown . <p > If both are null the no op filter will be returned . [CODESPLIT] public static Filter get ( FilterPredicate filterPredicate , UnboundRecordFilter unboundRecordFilter ) { checkArgument ( filterPredicate == null || unboundRecordFilter == null , "Cannot provide both a FilterPredicate and an UnboundRecordFilter" ) ; if ( filterPredicate != null ) { return get ( filterPredicate ) ; } if ( unboundRecordFilter != null ) { return get ( unboundRecordFilter ) ; } return NOOP ; }


Input: for files provided check if there s a summary file . If a summary file is found it is used otherwise the file footer is used . [CODESPLIT] @ Deprecated public static List < Footer > readAllFootersInParallelUsingSummaryFiles ( Configuration configuration , List < FileStatus > partFiles ) throws IOException { return readAllFootersInParallelUsingSummaryFiles ( configuration , partFiles , false ) ; }


Input: for files provided check if there s a summary file . If a summary file is found it is used otherwise the file footer is used . [CODESPLIT] @ Deprecated public static List < Footer > readAllFootersInParallelUsingSummaryFiles ( final Configuration configuration , final Collection < FileStatus > partFiles , final boolean skipRowGroups ) throws IOException { // figure out list of all parents to part files Set < Path > parents = new HashSet < Path > ( ) ; for ( FileStatus part : partFiles ) { parents . add ( part . getPath ( ) . getParent ( ) ) ; } // read corresponding summary files if they exist List < Callable < Map < Path , Footer > > > summaries = new ArrayList < Callable < Map < Path , Footer > > > ( ) ; for ( final Path path : parents ) { summaries . add ( new Callable < Map < Path , Footer > > ( ) { @ Override public Map < Path , Footer > call ( ) throws Exception { ParquetMetadata mergedMetadata = readSummaryMetadata ( configuration , path , skipRowGroups ) ; if ( mergedMetadata != null ) { final List < Footer > footers ; if ( skipRowGroups ) { footers = new ArrayList < Footer > ( ) ; for ( FileStatus f : partFiles ) { footers . add ( new Footer ( f . getPath ( ) , mergedMetadata ) ) ; } } else { footers = footersFromSummaryFile ( path , mergedMetadata ) ; } Map < Path , Footer > map = new HashMap < Path , Footer > ( ) ; for ( Footer footer : footers ) { // the folder may have been moved footer = new Footer ( new Path ( path , footer . getFile ( ) . getName ( ) ) , footer . getParquetMetadata ( ) ) ; map . put ( footer . getFile ( ) , footer ) ; } return map ; } else { return Collections . emptyMap ( ) ; } } } ) ; } Map < Path , Footer > cache = new HashMap < Path , Footer > ( ) ; try { List < Map < Path , Footer > > footersFromSummaries = runAllInParallel ( configuration . getInt ( PARQUET_READ_PARALLELISM , 5 ) , summaries ) ; for ( Map < Path , Footer > footers : footersFromSummaries ) { cache . putAll ( footers ) ; } } catch ( ExecutionException e ) { throw new IOException ( "Error reading summaries" , e ) ; } // keep only footers for files actually requested and read file footer if not found in summaries List < Footer > result = new ArrayList < Footer > ( partFiles . size ( ) ) ; List < FileStatus > toRead = new ArrayList < FileStatus > ( ) ; for ( FileStatus part : partFiles ) { Footer f = cache . get ( part . getPath ( ) ) ; if ( f != null ) { result . add ( f ) ; } else { toRead . add ( part ) ; } } if ( toRead . size ( ) > 0 ) { // read the footers of the files that did not have a summary file LOG . info ( "reading another {} footers" , toRead . size ( ) ) ; result . addAll ( readAllFootersInParallel ( configuration , toRead , skipRowGroups ) ) ; } return result ; }


Input: read all the footers of the files provided ( not using summary files ) [CODESPLIT] @ Deprecated public static List < Footer > readAllFootersInParallel ( final Configuration configuration , List < FileStatus > partFiles , final boolean skipRowGroups ) throws IOException { List < Callable < Footer >> footers = new ArrayList < Callable < Footer > > ( ) ; for ( final FileStatus currentFile : partFiles ) { footers . add ( new Callable < Footer > ( ) { @ Override public Footer call ( ) throws Exception { try { return new Footer ( currentFile . getPath ( ) , readFooter ( configuration , currentFile , filter ( skipRowGroups ) ) ) ; } catch ( IOException e ) { throw new IOException ( "Could not read footer for file " + currentFile , e ) ; } } } ) ; } try { return runAllInParallel ( configuration . getInt ( PARQUET_READ_PARALLELISM , 5 ) , footers ) ; } catch ( ExecutionException e ) { throw new IOException ( "Could not read footer: " + e . getMessage ( ) , e . getCause ( ) ) ; } }


Input: Read the footers of all the files under that path ( recursively ) not using summary files . [CODESPLIT] @ Deprecated public static List < Footer > readAllFootersInParallel ( Configuration configuration , FileStatus fileStatus , boolean skipRowGroups ) throws IOException { List < FileStatus > statuses = listFiles ( configuration , fileStatus ) ; return readAllFootersInParallel ( configuration , statuses , skipRowGroups ) ; }


Input: Read the footers of all the files under that path ( recursively ) not using summary files . rowGroups are not skipped [CODESPLIT] @ Deprecated public static List < Footer > readAllFootersInParallel ( Configuration configuration , FileStatus fileStatus ) throws IOException { return readAllFootersInParallel ( configuration , fileStatus , false ) ; }


Input: this always returns the row groups [CODESPLIT] @ Deprecated public static List < Footer > readFooters ( Configuration configuration , FileStatus pathStatus ) throws IOException { return readFooters ( configuration , pathStatus , false ) ; }


Input: Read the footers of all the files under that path ( recursively ) using summary files if possible [CODESPLIT] @ Deprecated public static List < Footer > readFooters ( Configuration configuration , FileStatus pathStatus , boolean skipRowGroups ) throws IOException { List < FileStatus > files = listFiles ( configuration , pathStatus ) ; return readAllFootersInParallelUsingSummaryFiles ( configuration , files , skipRowGroups ) ; }


Input: Specifically reads a given summary file [CODESPLIT] @ Deprecated public static List < Footer > readSummaryFile ( Configuration configuration , FileStatus summaryStatus ) throws IOException { final Path parent = summaryStatus . getPath ( ) . getParent ( ) ; ParquetMetadata mergedFooters = readFooter ( configuration , summaryStatus , filter ( false ) ) ; return footersFromSummaryFile ( parent , mergedFooters ) ; }


Input: Reads the meta data block in the footer of the file [CODESPLIT] @ Deprecated public static final ParquetMetadata readFooter ( Configuration configuration , Path file ) throws IOException { return readFooter ( configuration , file , NO_FILTER ) ; }


Input: Reads the meta data in the footer of the file . Skipping row groups ( or not ) based on the provided filter [CODESPLIT] public static ParquetMetadata readFooter ( Configuration configuration , Path file , MetadataFilter filter ) throws IOException { return readFooter ( HadoopInputFile . fromPath ( file , configuration ) , filter ) ; }


Input: Reads the meta data block in the footer of the file [CODESPLIT] @ Deprecated public static final ParquetMetadata readFooter ( Configuration configuration , FileStatus file , MetadataFilter filter ) throws IOException { return readFooter ( HadoopInputFile . fromStatus ( file , configuration ) , filter ) ; }


Input: Reads the meta data block in the footer of the file using provided input stream [CODESPLIT] @ Deprecated public static final ParquetMetadata readFooter ( InputFile file , MetadataFilter filter ) throws IOException { ParquetReadOptions options ; if ( file instanceof HadoopInputFile ) { options = HadoopReadOptions . builder ( ( ( HadoopInputFile ) file ) . getConfiguration ( ) ) . withMetadataFilter ( filter ) . build ( ) ; } else { options = ParquetReadOptions . builder ( ) . withMetadataFilter ( filter ) . build ( ) ; } try ( SeekableInputStream in = file . newStream ( ) ) { return readFooter ( file , options , in ) ; } }


Input: Open a { @link InputFile file } . [CODESPLIT] public static ParquetFileReader open ( InputFile file ) throws IOException { return new ParquetFileReader ( file , ParquetReadOptions . builder ( ) . build ( ) ) ; }


Input: Open a { @link InputFile file } with { @link ParquetReadOptions options } . [CODESPLIT] public static ParquetFileReader open ( InputFile file , ParquetReadOptions options ) throws IOException { return new ParquetFileReader ( file , options ) ; }


Input: Reads all the columns requested from the row group at the current file position . [CODESPLIT] public PageReadStore readNextRowGroup ( ) throws IOException { if ( currentBlock == blocks . size ( ) ) { return null ; } BlockMetaData block = blocks . get ( currentBlock ) ; if ( block . getRowCount ( ) == 0 ) { throw new RuntimeException ( "Illegal row group of 0 rows" ) ; } this . currentRowGroup = new ColumnChunkPageReadStore ( block . getRowCount ( ) ) ; // prepare the list of consecutive parts to read them in one scan List < ConsecutivePartList > allParts = new ArrayList < ConsecutivePartList > ( ) ; ConsecutivePartList currentParts = null ; for ( ColumnChunkMetaData mc : block . getColumns ( ) ) { ColumnPath pathKey = mc . getPath ( ) ; BenchmarkCounter . incrementTotalBytes ( mc . getTotalSize ( ) ) ; ColumnDescriptor columnDescriptor = paths . get ( pathKey ) ; if ( columnDescriptor != null ) { long startingPos = mc . getStartingPos ( ) ; // first part or not consecutive => new list if ( currentParts == null || currentParts . endPos ( ) != startingPos ) { currentParts = new ConsecutivePartList ( startingPos ) ; allParts . add ( currentParts ) ; } currentParts . addChunk ( new ChunkDescriptor ( columnDescriptor , mc , startingPos , ( int ) mc . getTotalSize ( ) ) ) ; } } // actually read all the chunks ChunkListBuilder builder = new ChunkListBuilder ( ) ; for ( ConsecutivePartList consecutiveChunks : allParts ) { consecutiveChunks . readAll ( f , builder ) ; } for ( Chunk chunk : builder . build ( ) ) { currentRowGroup . addColumn ( chunk . descriptor . col , chunk . readAllPages ( ) ) ; } // avoid re-reading bytes the dictionary reader is used after this call if ( nextDictionaryReader != null ) { nextDictionaryReader . setRowGroup ( currentRowGroup ) ; } advanceToNextBlock ( ) ; return currentRowGroup ; }


Input: Reads all the columns requested from the row group at the current file position . It may skip specific pages based on the column indexes according to the actual filter . As the rows are not aligned among the pages of the different columns row synchronization might be required . See the documentation of the class SynchronizingColumnReader for details . [CODESPLIT] public PageReadStore readNextFilteredRowGroup ( ) throws IOException { if ( currentBlock == blocks . size ( ) ) { return null ; } if ( ! options . useColumnIndexFilter ( ) ) { return readNextRowGroup ( ) ; } BlockMetaData block = blocks . get ( currentBlock ) ; if ( block . getRowCount ( ) == 0 ) { throw new RuntimeException ( "Illegal row group of 0 rows" ) ; } ColumnIndexStore ciStore = getColumnIndexStore ( currentBlock ) ; RowRanges rowRanges = getRowRanges ( currentBlock ) ; long rowCount = rowRanges . rowCount ( ) ; if ( rowCount == 0 ) { // There are no matching rows -> skipping this row-group advanceToNextBlock ( ) ; return readNextFilteredRowGroup ( ) ; } if ( rowCount == block . getRowCount ( ) ) { // All rows are matching -> fall back to the non-filtering path return readNextRowGroup ( ) ; } this . currentRowGroup = new ColumnChunkPageReadStore ( rowRanges ) ; // prepare the list of consecutive parts to read them in one scan ChunkListBuilder builder = new ChunkListBuilder ( ) ; List < ConsecutivePartList > allParts = new ArrayList < ConsecutivePartList > ( ) ; ConsecutivePartList currentParts = null ; for ( ColumnChunkMetaData mc : block . getColumns ( ) ) { ColumnPath pathKey = mc . getPath ( ) ; ColumnDescriptor columnDescriptor = paths . get ( pathKey ) ; if ( columnDescriptor != null ) { OffsetIndex offsetIndex = ciStore . getOffsetIndex ( mc . getPath ( ) ) ; OffsetIndex filteredOffsetIndex = filterOffsetIndex ( offsetIndex , rowRanges , block . getRowCount ( ) ) ; for ( OffsetRange range : calculateOffsetRanges ( filteredOffsetIndex , mc , offsetIndex . getOffset ( 0 ) ) ) { BenchmarkCounter . incrementTotalBytes ( range . getLength ( ) ) ; long startingPos = range . getOffset ( ) ; // first part or not consecutive => new list if ( currentParts == null || currentParts . endPos ( ) != startingPos ) { currentParts = new ConsecutivePartList ( startingPos ) ; allParts . add ( currentParts ) ; } ChunkDescriptor chunkDescriptor = new ChunkDescriptor ( columnDescriptor , mc , startingPos , ( int ) range . getLength ( ) ) ; currentParts . addChunk ( chunkDescriptor ) ; builder . setOffsetIndex ( chunkDescriptor , filteredOffsetIndex ) ; } } } // actually read all the chunks for ( ConsecutivePartList consecutiveChunks : allParts ) { consecutiveChunks . readAll ( f , builder ) ; } for ( Chunk chunk : builder . build ( ) ) { currentRowGroup . addColumn ( chunk . descriptor . col , chunk . readAllPages ( ) ) ; } // avoid re-reading bytes the dictionary reader is used after this call if ( nextDictionaryReader != null ) { nextDictionaryReader . setRowGroup ( currentRowGroup ) ; } advanceToNextBlock ( ) ; return currentRowGroup ; }


Input: Returns a { @link DictionaryPageReadStore } for the row group that would be returned by calling { @link #readNextRowGroup () } or skipped by calling { @link #skipNextRowGroup () } . [CODESPLIT] public DictionaryPageReadStore getNextDictionaryReader ( ) { if ( nextDictionaryReader == null && currentBlock < blocks . size ( ) ) { this . nextDictionaryReader = getDictionaryReader ( blocks . get ( currentBlock ) ) ; } return nextDictionaryReader ; }


Input: Reads and decompresses a dictionary page for the given column chunk . [CODESPLIT] DictionaryPage readDictionary ( ColumnChunkMetaData meta ) throws IOException { if ( ! meta . getEncodings ( ) . contains ( Encoding . PLAIN_DICTIONARY ) && ! meta . getEncodings ( ) . contains ( Encoding . RLE_DICTIONARY ) ) { return null ; } // TODO: this should use getDictionaryPageOffset() but it isn't reliable. if ( f . getPos ( ) != meta . getStartingPos ( ) ) { f . seek ( meta . getStartingPos ( ) ) ; } PageHeader pageHeader = Util . readPageHeader ( f ) ; if ( ! pageHeader . isSetDictionary_page_header ( ) ) { return null ; // TODO: should this complain? } DictionaryPage compressedPage = readCompressedDictionary ( pageHeader , f ) ; BytesInputDecompressor decompressor = options . getCodecFactory ( ) . getDecompressor ( meta . getCodec ( ) ) ; return new DictionaryPage ( decompressor . decompress ( compressedPage . getBytes ( ) , compressedPage . getUncompressedSize ( ) ) , compressedPage . getDictionarySize ( ) , compressedPage . getEncoding ( ) ) ; }


Input: Add a new writer and its memory allocation to the memory manager . [CODESPLIT] synchronized void addWriter ( InternalParquetRecordWriter writer , Long allocation ) { Long oldValue = writerList . get ( writer ) ; if ( oldValue == null ) { writerList . put ( writer , allocation ) ; } else { throw new IllegalArgumentException ( "[BUG] The Parquet Memory Manager should not add an " + "instance of InternalParquetRecordWriter more than once. The Manager already contains " + "the writer: " + writer ) ; } updateAllocation ( ) ; }


Input: Remove the given writer from the memory manager . [CODESPLIT] synchronized void removeWriter ( InternalParquetRecordWriter writer ) { if ( writerList . containsKey ( writer ) ) { writerList . remove ( writer ) ; } if ( ! writerList . isEmpty ( ) ) { updateAllocation ( ) ; } }


Input: Update the allocated size of each writer based on the current allocations and pool size . [CODESPLIT] private void updateAllocation ( ) { long totalAllocations = 0 ; for ( Long allocation : writerList . values ( ) ) { totalAllocations += allocation ; } if ( totalAllocations <= totalMemoryPool ) { scale = 1.0 ; } else { scale = ( double ) totalMemoryPool / totalAllocations ; LOG . warn ( String . format ( "Total allocation exceeds %.2f%% (%,d bytes) of heap memory\n" + "Scaling row group sizes to %.2f%% for %d writers" , 100 * memoryPoolRatio , totalMemoryPool , 100 * scale , writerList . size ( ) ) ) ; for ( Runnable callBack : callBacks . values ( ) ) { // we do not really want to start a new thread here. callBack . run ( ) ; } } int maxColCount = 0 ; for ( InternalParquetRecordWriter w : writerList . keySet ( ) ) { maxColCount = Math . max ( w . getSchema ( ) . getColumns ( ) . size ( ) , maxColCount ) ; } for ( Map . Entry < InternalParquetRecordWriter , Long > entry : writerList . entrySet ( ) ) { long newSize = ( long ) Math . floor ( entry . getValue ( ) * scale ) ; if ( scale < 1.0 && minMemoryAllocation > 0 && newSize < minMemoryAllocation ) { throw new ParquetRuntimeException ( String . format ( "New Memory allocation %d bytes" + " is smaller than the minimum allocation size of %d bytes." , newSize , minMemoryAllocation ) ) { } ; } entry . getKey ( ) . setRowGroupSizeThreshold ( newSize ) ; LOG . debug ( String . format ( "Adjust block size from %,d to %,d for writer: %s" , entry . getValue ( ) , newSize , entry . getKey ( ) ) ) ; } }


Input: Register callback and deduplicate it if any . [CODESPLIT] public void registerScaleCallBack ( String callBackName , Runnable callBack ) { Preconditions . checkNotNull ( callBackName , "callBackName" ) ; Preconditions . checkNotNull ( callBack , "callBack" ) ; if ( callBacks . containsKey ( callBackName ) ) { throw new IllegalArgumentException ( "The callBackName " + callBackName + " is duplicated and has been registered already." ) ; } else { callBacks . put ( callBackName , callBack ) ; } }


Input: start the file [CODESPLIT] public void start ( ) throws IOException { state = state . start ( ) ; LOG . debug ( "{}: start" , out . getPos ( ) ) ; out . write ( MAGIC ) ; }


Input: start a block [CODESPLIT] public void startBlock ( long recordCount ) throws IOException { state = state . startBlock ( ) ; LOG . debug ( "{}: start block" , out . getPos ( ) ) ; //    out.write(MAGIC); // TODO: add a magic delimiter alignment . alignForRowGroup ( out ) ; currentBlock = new BlockMetaData ( ) ; currentRecordCount = recordCount ; currentColumnIndexes = new ArrayList <> ( ) ; currentOffsetIndexes = new ArrayList <> ( ) ; }


Input: start a column inside a block [CODESPLIT] public void startColumn ( ColumnDescriptor descriptor , long valueCount , CompressionCodecName compressionCodecName ) throws IOException { state = state . startColumn ( ) ; encodingStatsBuilder . clear ( ) ; currentEncodings = new HashSet < Encoding > ( ) ; currentChunkPath = ColumnPath . get ( descriptor . getPath ( ) ) ; currentChunkType = descriptor . getPrimitiveType ( ) ; currentChunkCodec = compressionCodecName ; currentChunkValueCount = valueCount ; currentChunkFirstDataPage = out . getPos ( ) ; compressedLength = 0 ; uncompressedLength = 0 ; // The statistics will be copied from the first one added at writeDataPage(s) so we have the correct typed one currentStatistics = null ; columnIndexBuilder = ColumnIndexBuilder . getBuilder ( currentChunkType , columnIndexTruncateLength ) ; offsetIndexBuilder = OffsetIndexBuilder . getBuilder ( ) ; firstPageOffset = - 1 ; }


Input: writes a dictionary page page [CODESPLIT] public void writeDictionaryPage ( DictionaryPage dictionaryPage ) throws IOException { state = state . write ( ) ; LOG . debug ( "{}: write dictionary page: {} values" , out . getPos ( ) , dictionaryPage . getDictionarySize ( ) ) ; currentChunkDictionaryPageOffset = out . getPos ( ) ; int uncompressedSize = dictionaryPage . getUncompressedSize ( ) ; int compressedPageSize = ( int ) dictionaryPage . getBytes ( ) . size ( ) ; // TODO: fix casts metadataConverter . writeDictionaryPageHeader ( uncompressedSize , compressedPageSize , dictionaryPage . getDictionarySize ( ) , dictionaryPage . getEncoding ( ) , out ) ; long headerSize = out . getPos ( ) - currentChunkDictionaryPageOffset ; this . uncompressedLength += uncompressedSize + headerSize ; this . compressedLength += compressedPageSize + headerSize ; LOG . debug ( "{}: write dictionary page content {}" , out . getPos ( ) , compressedPageSize ) ; dictionaryPage . getBytes ( ) . writeAllTo ( out ) ; encodingStatsBuilder . addDictEncoding ( dictionaryPage . getEncoding ( ) ) ; currentEncodings . add ( dictionaryPage . getEncoding ( ) ) ; }


Input: writes a single page [CODESPLIT] @ Deprecated public void writeDataPage ( int valueCount , int uncompressedPageSize , BytesInput bytes , Encoding rlEncoding , Encoding dlEncoding , Encoding valuesEncoding ) throws IOException { state = state . write ( ) ; // We are unable to build indexes without rowCount so skip them for this column offsetIndexBuilder = OffsetIndexBuilder . getNoOpBuilder ( ) ; columnIndexBuilder = ColumnIndexBuilder . getNoOpBuilder ( ) ; long beforeHeader = out . getPos ( ) ; LOG . debug ( "{}: write data page: {} values" , beforeHeader , valueCount ) ; int compressedPageSize = ( int ) bytes . size ( ) ; metadataConverter . writeDataPageV1Header ( uncompressedPageSize , compressedPageSize , valueCount , rlEncoding , dlEncoding , valuesEncoding , out ) ; long headerSize = out . getPos ( ) - beforeHeader ; this . uncompressedLength += uncompressedPageSize + headerSize ; this . compressedLength += compressedPageSize + headerSize ; LOG . debug ( "{}: write data page content {}" , out . getPos ( ) , compressedPageSize ) ; bytes . writeAllTo ( out ) ; encodingStatsBuilder . addDataEncoding ( valuesEncoding ) ; currentEncodings . add ( rlEncoding ) ; currentEncodings . add ( dlEncoding ) ; currentEncodings . add ( valuesEncoding ) ; }


Input: writes a single page [CODESPLIT] @ Deprecated public void writeDataPage ( int valueCount , int uncompressedPageSize , BytesInput bytes , Statistics statistics , Encoding rlEncoding , Encoding dlEncoding , Encoding valuesEncoding ) throws IOException { // We are unable to build indexes without rowCount so skip them for this column offsetIndexBuilder = OffsetIndexBuilder . getNoOpBuilder ( ) ; columnIndexBuilder = ColumnIndexBuilder . getNoOpBuilder ( ) ; innerWriteDataPage ( valueCount , uncompressedPageSize , bytes , statistics , rlEncoding , dlEncoding , valuesEncoding ) ; }


Input: Writes a single page [CODESPLIT] public void writeDataPage ( int valueCount , int uncompressedPageSize , BytesInput bytes , Statistics statistics , long rowCount , Encoding rlEncoding , Encoding dlEncoding , Encoding valuesEncoding ) throws IOException { long beforeHeader = out . getPos ( ) ; innerWriteDataPage ( valueCount , uncompressedPageSize , bytes , statistics , rlEncoding , dlEncoding , valuesEncoding ) ; offsetIndexBuilder . add ( ( int ) ( out . getPos ( ) - beforeHeader ) , rowCount ) ; }


Input: Writes a column chunk at once [CODESPLIT] void writeColumnChunk ( ColumnDescriptor descriptor , long valueCount , CompressionCodecName compressionCodecName , DictionaryPage dictionaryPage , BytesInput bytes , long uncompressedTotalPageSize , long compressedTotalPageSize , Statistics < ? > totalStats , ColumnIndexBuilder columnIndexBuilder , OffsetIndexBuilder offsetIndexBuilder , Set < Encoding > rlEncodings , Set < Encoding > dlEncodings , List < Encoding > dataEncodings ) throws IOException { startColumn ( descriptor , valueCount , compressionCodecName ) ; state = state . write ( ) ; if ( dictionaryPage != null ) { writeDictionaryPage ( dictionaryPage ) ; } LOG . debug ( "{}: write data pages" , out . getPos ( ) ) ; long headersSize = bytes . size ( ) - compressedTotalPageSize ; this . uncompressedLength += uncompressedTotalPageSize + headersSize ; this . compressedLength += compressedTotalPageSize + headersSize ; LOG . debug ( "{}: write data pages content" , out . getPos ( ) ) ; firstPageOffset = out . getPos ( ) ; bytes . writeAllTo ( out ) ; encodingStatsBuilder . addDataEncodings ( dataEncodings ) ; if ( rlEncodings . isEmpty ( ) ) { encodingStatsBuilder . withV2Pages ( ) ; } currentEncodings . addAll ( rlEncodings ) ; currentEncodings . addAll ( dlEncodings ) ; currentEncodings . addAll ( dataEncodings ) ; currentStatistics = totalStats ; this . columnIndexBuilder = columnIndexBuilder ; this . offsetIndexBuilder = offsetIndexBuilder ; endColumn ( ) ; }


Input: end a column ( once all rep def and data have been written ) [CODESPLIT] public void endColumn ( ) throws IOException { state = state . endColumn ( ) ; LOG . debug ( "{}: end column" , out . getPos ( ) ) ; if ( columnIndexBuilder . getMinMaxSize ( ) > columnIndexBuilder . getPageCount ( ) * MAX_STATS_SIZE ) { currentColumnIndexes . add ( null ) ; } else { currentColumnIndexes . add ( columnIndexBuilder . build ( ) ) ; } currentOffsetIndexes . add ( offsetIndexBuilder . build ( firstPageOffset ) ) ; currentBlock . addColumn ( ColumnChunkMetaData . get ( currentChunkPath , currentChunkType , currentChunkCodec , encodingStatsBuilder . build ( ) , currentEncodings , currentStatistics , currentChunkFirstDataPage , currentChunkDictionaryPageOffset , currentChunkValueCount , compressedLength , uncompressedLength ) ) ; this . currentBlock . setTotalByteSize ( currentBlock . getTotalByteSize ( ) + uncompressedLength ) ; this . uncompressedLength = 0 ; this . compressedLength = 0 ; columnIndexBuilder = null ; offsetIndexBuilder = null ; }


Input: ends a block once all column chunks have been written [CODESPLIT] public void endBlock ( ) throws IOException { state = state . endBlock ( ) ; LOG . debug ( "{}: end block" , out . getPos ( ) ) ; currentBlock . setRowCount ( currentRecordCount ) ; blocks . add ( currentBlock ) ; columnIndexes . add ( currentColumnIndexes ) ; offsetIndexes . add ( currentOffsetIndexes ) ; currentColumnIndexes = null ; currentOffsetIndexes = null ; currentBlock = null ; }


Input: Copy from a FS input stream to an output stream . Thread - safe [CODESPLIT] private static void copy ( SeekableInputStream from , PositionOutputStream to , long start , long length ) throws IOException { LOG . debug ( "Copying {} bytes at {} to {}" , length , start , to . getPos ( ) ) ; from . seek ( start ) ; long bytesCopied = 0 ; byte [ ] buffer = COPY_BUFFER . get ( ) ; while ( bytesCopied < length ) { long bytesLeft = length - bytesCopied ; int bytesRead = from . read ( buffer , 0 , ( buffer . length < bytesLeft ? buffer . length : ( int ) bytesLeft ) ) ; if ( bytesRead < 0 ) { throw new IllegalArgumentException ( "Unexpected end of input file at " + start + bytesCopied ) ; } to . write ( buffer , 0 , bytesRead ) ; bytesCopied += bytesRead ; } }


Input: ends a file once all blocks have been written . closes the file . [CODESPLIT] public void end ( Map < String , String > extraMetaData ) throws IOException { state = state . end ( ) ; serializeColumnIndexes ( columnIndexes , blocks , out ) ; serializeOffsetIndexes ( offsetIndexes , blocks , out ) ; LOG . debug ( "{}: end" , out . getPos ( ) ) ; this . footer = new ParquetMetadata ( new FileMetaData ( schema , extraMetaData , Version . FULL_VERSION ) , blocks ) ; serializeFooter ( footer , out ) ; out . close ( ) ; }


Input: Given a list of metadata files merge them into a single ParquetMetadata Requires that the schemas be compatible and the extraMetadata be exactly equal . [CODESPLIT] @ Deprecated public static ParquetMetadata mergeMetadataFiles ( List < Path > files , Configuration conf ) throws IOException { Preconditions . checkArgument ( ! files . isEmpty ( ) , "Cannot merge an empty list of metadata" ) ; GlobalMetaData globalMetaData = null ; List < BlockMetaData > blocks = new ArrayList < BlockMetaData > ( ) ; for ( Path p : files ) { ParquetMetadata pmd = ParquetFileReader . readFooter ( conf , p , ParquetMetadataConverter . NO_FILTER ) ; FileMetaData fmd = pmd . getFileMetaData ( ) ; globalMetaData = mergeInto ( fmd , globalMetaData , true ) ; blocks . addAll ( pmd . getBlocks ( ) ) ; } // collapse GlobalMetaData into a single FileMetaData, which will throw if they are not compatible return new ParquetMetadata ( globalMetaData . merge ( ) , blocks ) ; }


Input: Given a list of metadata files merge them into a single metadata file . Requires that the schemas be compatible and the extraMetaData be exactly equal . This is useful when merging 2 directories of parquet files into a single directory as long as both directories were written with compatible schemas and equal extraMetaData . [CODESPLIT] @ Deprecated public static void writeMergedMetadataFile ( List < Path > files , Path outputPath , Configuration conf ) throws IOException { ParquetMetadata merged = mergeMetadataFiles ( files , conf ) ; writeMetadataFile ( outputPath , merged , outputPath . getFileSystem ( conf ) ) ; }


Input: writes a _metadata and _common_metadata file [CODESPLIT] @ Deprecated public static void writeMetadataFile ( Configuration configuration , Path outputPath , List < Footer > footers ) throws IOException { writeMetadataFile ( configuration , outputPath , footers , JobSummaryLevel . ALL ) ; }


Input: writes _common_metadata file and optionally a _metadata file depending on the { [CODESPLIT] @ Deprecated public static void writeMetadataFile ( Configuration configuration , Path outputPath , List < Footer > footers , JobSummaryLevel level ) throws IOException { Preconditions . checkArgument ( level == JobSummaryLevel . ALL || level == JobSummaryLevel . COMMON_ONLY , "Unsupported level: " + level ) ; FileSystem fs = outputPath . getFileSystem ( configuration ) ; outputPath = outputPath . makeQualified ( fs ) ; ParquetMetadata metadataFooter = mergeFooters ( outputPath , footers ) ; if ( level == JobSummaryLevel . ALL ) { writeMetadataFile ( outputPath , metadataFooter , fs , PARQUET_METADATA_FILE ) ; } metadataFooter . getBlocks ( ) . clear ( ) ; writeMetadataFile ( outputPath , metadataFooter , fs , PARQUET_COMMON_METADATA_FILE ) ; }


Input: Will return the result of merging toMerge into mergedMetadata [CODESPLIT] static GlobalMetaData mergeInto ( FileMetaData toMerge , GlobalMetaData mergedMetadata ) { return mergeInto ( toMerge , mergedMetadata , true ) ; }


Input: will return the result of merging toMerge into mergedSchema [CODESPLIT] static MessageType mergeInto ( MessageType toMerge , MessageType mergedSchema ) { return mergeInto ( toMerge , mergedSchema , true ) ; }


Input: will return the result of merging toMerge into mergedSchema [CODESPLIT] static MessageType mergeInto ( MessageType toMerge , MessageType mergedSchema , boolean strict ) { if ( mergedSchema == null ) { return toMerge ; } return mergedSchema . union ( toMerge , strict ) ; }


Input: Reads the value into the binding . [CODESPLIT] public void readValue ( ) { try { if ( ! valueRead ) { binding . read ( ) ; valueRead = true ; } } catch ( RuntimeException e ) { if ( CorruptDeltaByteArrays . requiresSequentialReads ( writerVersion , currentEncoding ) && e instanceof ArrayIndexOutOfBoundsException ) { // this is probably PARQUET-246, which may happen if reading data with // MR because this can't be detected without reading all footers throw new ParquetDecodingException ( "Read failure possibly due to " + "PARQUET-246: try setting parquet.split.files to false" , new ParquetDecodingException ( format ( "Can't read value in column %s at value %d out of %d, " + "%d out of %d in currentPage. repetition level: " + "%d, definition level: %d" , path , readValues , totalValueCount , readValues - ( endOfPageValueCount - pageValueCount ) , pageValueCount , repetitionLevel , definitionLevel ) , e ) ) ; } throw new ParquetDecodingException ( format ( "Can't read value in column %s at value %d out of %d, " + "%d out of %d in currentPage. repetition level: " + "%d, definition level: %d" , path , readValues , totalValueCount , readValues - ( endOfPageValueCount - pageValueCount ) , pageValueCount , repetitionLevel , definitionLevel ) , e ) ; } }


Input: Returns whether null is allowed by the schema . [CODESPLIT] public static boolean nullOk ( Schema schema ) { if ( Schema . Type . NULL == schema . getType ( ) ) { return true ; } else if ( Schema . Type . UNION == schema . getType ( ) ) { for ( Schema possible : schema . getTypes ( ) ) { if ( nullOk ( possible ) ) { return true ; } } } return false ; }


Input: Merges { @link Schema } instances if they are compatible . <p > Schemas are incompatible if : <ul > <li > The { @link Schema . Type } does not match . < / li > <li > For record schemas the record name does not match< / li > <li > For enum schemas the enum name does not match< / li > < / ul > <p > Map value array element and record field types types will use unions if necessary and union schemas are merged recursively . [CODESPLIT] public static Schema merge ( Iterable < Schema > schemas ) { Iterator < Schema > iter = schemas . iterator ( ) ; if ( ! iter . hasNext ( ) ) { return null ; } Schema result = iter . next ( ) ; while ( iter . hasNext ( ) ) { result = merge ( result , iter . next ( ) ) ; } return result ; }


Input: Merges two { @link Schema } instances if they are compatible . <p > Two schemas are incompatible if : <ul > <li > The { @link Schema . Type } does not match . < / li > <li > For record schemas the record name does not match< / li > <li > For enum schemas the enum name does not match< / li > < / ul > <p > Map value and array element types will use unions if necessary and union schemas are merged recursively . [CODESPLIT] public static Schema merge ( Schema left , Schema right ) { Schema merged = mergeOnly ( left , right ) ; Preconditions . checkState ( merged != null , "Cannot merge %s and %s" , left , right ) ; return merged ; }


Input: Merges two { @link Schema } instances or returns { @code null } . <p > The two schemas are merged if they are the same type . Records are merged if the two records have the same name or have no names but have a significant number of shared fields . <p > @see { @link #mergeOrUnion } to return a union when a merge is not possible . [CODESPLIT] private static Schema mergeOrUnion ( Schema left , Schema right ) { Schema merged = mergeOnly ( left , right ) ; if ( merged != null ) { return merged ; } return union ( left , right ) ; }


Input: Creates a union of two { @link Schema } instances . <p > If either { @code Schema } is a union this will attempt to merge the other schema with the types contained in that union before adding more types to the union that is produced . <p > If both schemas are not unions no merge is attempted . [CODESPLIT] private static Schema union ( Schema left , Schema right ) { if ( left . getType ( ) == Schema . Type . UNION ) { if ( right . getType ( ) == Schema . Type . UNION ) { // combine the unions by adding each type in right individually Schema combined = left ; for ( Schema type : right . getTypes ( ) ) { combined = union ( combined , type ) ; } return combined ; } else { boolean notMerged = true ; // combine a union with a non-union by checking if each type will merge List < Schema > types = Lists . newArrayList ( ) ; Iterator < Schema > schemas = left . getTypes ( ) . iterator ( ) ; // try to merge each type and stop when one succeeds while ( schemas . hasNext ( ) ) { Schema next = schemas . next ( ) ; Schema merged = mergeOnly ( next , right ) ; if ( merged != null ) { types . add ( merged ) ; notMerged = false ; break ; } else { // merge didn't work, add the type types . add ( next ) ; } } // add the remaining types from the left union while ( schemas . hasNext ( ) ) { types . add ( schemas . next ( ) ) ; } if ( notMerged ) { types . add ( right ) ; } return Schema . createUnion ( types ) ; } } else if ( right . getType ( ) == Schema . Type . UNION ) { return union ( right , left ) ; } return Schema . createUnion ( ImmutableList . of ( left , right ) ) ; }


Input: Merges two { @link Schema } instances or returns { @code null } . <p > The two schemas are merged if they are the same type . Records are merged if the two records have the same name or have no names but have a significant number of shared fields . <p > @see { @link #mergeOrUnion } to return a union when a merge is not possible . [CODESPLIT] private static Schema mergeOnly ( Schema left , Schema right ) { if ( Objects . equal ( left , right ) ) { return left ; } // handle primitive type promotion; doesn't promote integers to floats switch ( left . getType ( ) ) { case INT : if ( right . getType ( ) == Schema . Type . LONG ) { return right ; } break ; case LONG : if ( right . getType ( ) == Schema . Type . INT ) { return left ; } break ; case FLOAT : if ( right . getType ( ) == Schema . Type . DOUBLE ) { return right ; } break ; case DOUBLE : if ( right . getType ( ) == Schema . Type . FLOAT ) { return left ; } } // any other cases where the types don't match must be combined by a union if ( left . getType ( ) != right . getType ( ) ) { return null ; } switch ( left . getType ( ) ) { case UNION : return union ( left , right ) ; case RECORD : if ( left . getName ( ) == null && right . getName ( ) == null && fieldSimilarity ( left , right ) < SIMILARITY_THRESH ) { return null ; } else if ( ! Objects . equal ( left . getName ( ) , right . getName ( ) ) ) { return null ; } Schema combinedRecord = Schema . createRecord ( coalesce ( left . getName ( ) , right . getName ( ) ) , coalesce ( left . getDoc ( ) , right . getDoc ( ) ) , coalesce ( left . getNamespace ( ) , right . getNamespace ( ) ) , false ) ; combinedRecord . setFields ( mergeFields ( left , right ) ) ; return combinedRecord ; case MAP : return Schema . createMap ( mergeOrUnion ( left . getValueType ( ) , right . getValueType ( ) ) ) ; case ARRAY : return Schema . createArray ( mergeOrUnion ( left . getElementType ( ) , right . getElementType ( ) ) ) ; case ENUM : if ( ! Objects . equal ( left . getName ( ) , right . getName ( ) ) ) { return null ; } Set < String > symbols = Sets . newLinkedHashSet ( ) ; symbols . addAll ( left . getEnumSymbols ( ) ) ; symbols . addAll ( right . getEnumSymbols ( ) ) ; return Schema . createEnum ( left . getName ( ) , coalesce ( left . getDoc ( ) , right . getDoc ( ) ) , coalesce ( left . getNamespace ( ) , right . getNamespace ( ) ) , ImmutableList . copyOf ( symbols ) ) ; default : // all primitives are handled before the switch by the equality check. // schemas that reach this point are not primitives and also not any of // the above known types. throw new UnsupportedOperationException ( "Unknown schema type: " + left . getType ( ) ) ; } }


Input: Returns a union { @link Schema } of NULL and the given { @code schema } . <p > A NULL schema is always the first type in the union so that a null default value can be set . [CODESPLIT] private static Schema nullableForDefault ( Schema schema ) { if ( schema . getType ( ) == Schema . Type . NULL ) { return schema ; } if ( schema . getType ( ) != Schema . Type . UNION ) { return Schema . createUnion ( ImmutableList . of ( NULL , schema ) ) ; } if ( schema . getTypes ( ) . get ( 0 ) . getType ( ) == Schema . Type . NULL ) { return schema ; } List < Schema > types = Lists . newArrayList ( ) ; types . add ( NULL ) ; for ( Schema type : schema . getTypes ( ) ) { if ( type . getType ( ) != Schema . Type . NULL ) { types . add ( type ) ; } } return Schema . createUnion ( types ) ; }


Input: Creates a new field with the same name schema doc and default value as the incoming schema . <p > Fields cannot be used in more than one record ( not Immutable? ) . [CODESPLIT] public static Schema . Field copy ( Schema . Field field ) { return new Schema . Field ( field . name ( ) , field . schema ( ) , field . doc ( ) , field . defaultValue ( ) ) ; }


Input: Returns the first non - null object that is passed in . [CODESPLIT] @ SafeVarargs private static < E > E coalesce ( E ... objects ) { for ( E object : objects ) { if ( object != null ) { return object ; } } return null ; }


Input: { [CODESPLIT] @ Override public void close ( TaskAttemptContext context ) throws IOException , InterruptedException { try { internalWriter . close ( ) ; // release after the writer closes in case it is used for a last flush } finally { if ( codecFactory != null ) { codecFactory . release ( ) ; } if ( memoryManager != null ) { memoryManager . removeWriter ( internalWriter ) ; } } }


Input: { [CODESPLIT] @ Override public void write ( Void key , T value ) throws IOException , InterruptedException { internalWriter . write ( value ) ; }


Input: Returns a the value as the first matching schema type or null . [CODESPLIT] private static Object makeValue ( String string , Schema schema ) { if ( string == null ) { return null ; } try { switch ( schema . getType ( ) ) { case BOOLEAN : return Boolean . valueOf ( string ) ; case STRING : return string ; case FLOAT : return Float . valueOf ( string ) ; case DOUBLE : return Double . valueOf ( string ) ; case INT : return Integer . valueOf ( string ) ; case LONG : return Long . valueOf ( string ) ; case ENUM : // TODO: translate to enum class if ( schema . hasEnumSymbol ( string ) ) { return string ; } else { try { return schema . getEnumSymbols ( ) . get ( Integer . parseInt ( string ) ) ; } catch ( IndexOutOfBoundsException ex ) { return null ; } } case UNION : Object value = null ; for ( Schema possible : schema . getTypes ( ) ) { value = makeValue ( string , possible ) ; if ( value != null ) { return value ; } } return null ; case NULL : return null ; default : // FIXED, BYTES, MAP, ARRAY, RECORD are not supported throw new RecordException ( "Unsupported field type:" + schema . getType ( ) ) ; } } catch ( NumberFormatException e ) { // empty string is considered null for numeric types if ( string . isEmpty ( ) ) { return null ; } else { throw e ; } } }


Input: Reads ThriftMetadata from the parquet file footer . [CODESPLIT] public static ThriftMetaData fromExtraMetaData ( Map < String , String > extraMetaData ) { final String thriftClassName = extraMetaData . get ( THRIFT_CLASS ) ; final String thriftDescriptorString = extraMetaData . get ( THRIFT_DESCRIPTOR ) ; if ( thriftClassName == null || thriftDescriptorString == null ) { return null ; } final StructType descriptor = parseDescriptor ( thriftDescriptorString ) ; return new ThriftMetaData ( thriftClassName , descriptor ) ; }


Input: Creates ThriftMetaData from a Thrift - generated class . [CODESPLIT] @ SuppressWarnings ( "unchecked" ) public static ThriftMetaData fromThriftClass ( Class < ? > thriftClass ) { if ( thriftClass != null && TBase . class . isAssignableFrom ( thriftClass ) ) { Class < ? extends TBase < ? , ? > > tClass = ( Class < ? extends TBase < ? , ? > > ) thriftClass ; StructType descriptor = new ThriftSchemaConverter ( ) . toStructType ( tClass ) ; return new ThriftMetaData ( thriftClass . getName ( ) , descriptor ) ; } return null ; }


Input: generates a map of key values to store in the footer [CODESPLIT] public Map < String , String > toExtraMetaData ( ) { final Map < String , String > map = new HashMap < String , String > ( ) ; map . put ( THRIFT_CLASS , getThriftClass ( ) . getName ( ) ) ; map . put ( THRIFT_DESCRIPTOR , descriptor . toJSON ( ) ) ; return map ; }


Input: Writes the current null value [CODESPLIT] @ Override public void writeNull ( int repetitionLevel , int definitionLevel ) { if ( DEBUG ) log ( null , repetitionLevel , definitionLevel ) ; repetitionLevel ( repetitionLevel ) ; definitionLevel ( definitionLevel ) ; statistics . incrementNumNulls ( ) ; ++ valueCount ; }


Input: Writes the current value [CODESPLIT] @ Override public void write ( double value , int repetitionLevel , int definitionLevel ) { if ( DEBUG ) log ( value , repetitionLevel , definitionLevel ) ; repetitionLevel ( repetitionLevel ) ; definitionLevel ( definitionLevel ) ; dataColumn . writeDouble ( value ) ; statistics . updateStats ( value ) ; ++ valueCount ; }


Input: Writes the current value [CODESPLIT] @ Override public void write ( Binary value , int repetitionLevel , int definitionLevel ) { if ( DEBUG ) log ( value , repetitionLevel , definitionLevel ) ; repetitionLevel ( repetitionLevel ) ; definitionLevel ( definitionLevel ) ; dataColumn . writeBytes ( value ) ; statistics . updateStats ( value ) ; ++ valueCount ; }


Input: Finalizes the Column chunk . Possibly adding extra pages if needed ( dictionary ... ) Is called right after writePage [CODESPLIT] void finalizeColumnChunk ( ) { final DictionaryPage dictionaryPage = dataColumn . toDictPageAndClose ( ) ; if ( dictionaryPage != null ) { if ( DEBUG ) LOG . debug ( "write dictionary" ) ; try { pageWriter . writeDictionaryPage ( dictionaryPage ) ; } catch ( IOException e ) { throw new ParquetEncodingException ( "could not write dictionary page for " + path , e ) ; } dataColumn . resetDictionary ( ) ; } }


Input: Writes the current data to a new page in the page store [CODESPLIT] void writePage ( ) { if ( valueCount == 0 ) { throw new ParquetEncodingException ( "writing empty page" ) ; } this . rowsWrittenSoFar += pageRowCount ; if ( DEBUG ) LOG . debug ( "write page" ) ; try { writePage ( pageRowCount , valueCount , statistics , repetitionLevelColumn , definitionLevelColumn , dataColumn ) ; } catch ( IOException e ) { throw new ParquetEncodingException ( "could not write page for " + path , e ) ; } repetitionLevelColumn . reset ( ) ; definitionLevelColumn . reset ( ) ; dataColumn . reset ( ) ; valueCount = 0 ; resetStatistics ( ) ; pageRowCount = 0 ; }


Input: visible for testing [CODESPLIT] static List < String > parseSemicolonDelimitedString ( String columnsToKeepGlobs ) { String [ ] splits = columnsToKeepGlobs . split ( GLOB_SEPARATOR ) ; List < String > globs = new ArrayList < String > ( ) ; for ( String s : splits ) { if ( ! s . isEmpty ( ) ) { globs . add ( s ) ; } } if ( globs . isEmpty ( ) ) { throw new ThriftProjectionException ( String . format ( "Semicolon delimited string '%s' contains 0 glob strings" , columnsToKeepGlobs ) ) ; } return globs ; }


Input: visible for testing [CODESPLIT] boolean keep ( String path ) { WildcardPath match = null ; // since we have a rule of every path must match at least one column, // we visit every single wildcard path, instead of short circuiting, // for the case where more than one pattern matches a column. Otherwise // we'd get a misleading exception saying a path didn't match a column, // even though it looks like it should have (but didn't because of short circuiting). // This also allows us log a warning when more than one glob path matches. for ( WildcardPathStatus wp : columnsToKeep ) { if ( wp . matches ( path ) ) { if ( match != null && ! match . getParentGlobPath ( ) . equals ( wp . getWildcardPath ( ) . getParentGlobPath ( ) ) ) { String message = "Field path: '%s' matched more than one glob path pattern. First match: " + "'%s' (when expanded to '%s') second match:'%s' (when expanded to '%s')" ; warn ( String . format ( message , path , match . getParentGlobPath ( ) , match . getOriginalPattern ( ) , wp . getWildcardPath ( ) . getParentGlobPath ( ) , wp . getWildcardPath ( ) . getOriginalPattern ( ) ) ) ; } else { match = wp . getWildcardPath ( ) ; } } } return match != null ; }


Input: { [CODESPLIT] public void endMessage ( ) { delegate . endMessage ( ) ; validateMissingFields ( types . peek ( ) . asGroupType ( ) . getFieldCount ( ) ) ; previousField . pop ( ) ; }


Input: { [CODESPLIT] public void startField ( String field , int index ) { if ( index <= previousField . peek ( ) ) { throw new InvalidRecordException ( "fields must be added in order " + field + " index " + index + " is before previous field " + previousField . peek ( ) ) ; } validateMissingFields ( index ) ; fields . push ( index ) ; fieldValueCount . push ( 0 ) ; delegate . startField ( field , index ) ; }


Input: { [CODESPLIT] public void endField ( String field , int index ) { delegate . endField ( field , index ) ; fieldValueCount . pop ( ) ; previousField . push ( fields . pop ( ) ) ; }


Input: { [CODESPLIT] public void startGroup ( ) { previousField . push ( - 1 ) ; types . push ( types . peek ( ) . asGroupType ( ) . getType ( fields . peek ( ) ) ) ; delegate . startGroup ( ) ; }


Input: { [CODESPLIT] public void endGroup ( ) { delegate . endGroup ( ) ; validateMissingFields ( types . peek ( ) . asGroupType ( ) . getFieldCount ( ) ) ; types . pop ( ) ; previousField . pop ( ) ; }


Input: { [CODESPLIT] public void addBinary ( Binary value ) { validate ( BINARY , INT96 , FIXED_LEN_BYTE_ARRAY ) ; delegate . addBinary ( value ) ; }


Input: { [CODESPLIT] @ Override public void writeToStringBuilder ( StringBuilder sb , String indent ) { sb . append ( indent ) . append ( getRepetition ( ) . name ( ) . toLowerCase ( Locale . ENGLISH ) ) . append ( " " ) . append ( primitive . name ( ) . toLowerCase ( ) ) ; if ( primitive == PrimitiveTypeName . FIXED_LEN_BYTE_ARRAY ) { sb . append ( "(" + length + ")" ) ; } sb . append ( " " ) . append ( getName ( ) ) ; if ( getLogicalTypeAnnotation ( ) != null ) { // TODO: should we print decimal metadata too? sb . append ( " (" ) . append ( getLogicalTypeAnnotation ( ) . toString ( ) ) . append ( ")" ) ; } if ( getId ( ) != null ) { sb . append ( " = " ) . append ( getId ( ) ) ; } }


Input: Returns the { @link Type } specific comparator for properly comparing values . The natural ordering of the values might not proper in certain cases ( e . g . { @code UINT_32 } requires unsigned comparison of { @code int } values while the natural ordering is signed . ) [CODESPLIT] @ SuppressWarnings ( "unchecked" ) public < T > PrimitiveComparator < T > comparator ( ) { return ( PrimitiveComparator < T > ) getPrimitiveTypeName ( ) . comparator ( getLogicalTypeAnnotation ( ) ) ; }


Input: eagerly loads all the data into memory [CODESPLIT] @ Override public void initFromPage ( int valueCount , ByteBufferInputStream stream ) throws IOException { this . in = stream ; long startPos = in . position ( ) ; this . config = DeltaBinaryPackingConfig . readConfig ( in ) ; this . totalValueCount = BytesUtils . readUnsignedVarInt ( in ) ; allocateValuesBuffer ( ) ; bitWidths = new int [ config . miniBlockNumInABlock ] ; //read first value from header valuesBuffer [ valuesBuffered ++ ] = BytesUtils . readZigZagVarLong ( in ) ; while ( valuesBuffered < totalValueCount ) { //values Buffered could be more than totalValueCount, since we flush on a mini block basis loadNewBlockToBuffer ( ) ; } updateNextOffset ( ( int ) ( in . position ( ) - startPos ) ) ; }


Input: the value buffer is allocated so that the size of it is multiple of mini block because when writing data is flushed on a mini block basis [CODESPLIT] private void allocateValuesBuffer ( ) { int totalMiniBlockCount = ( int ) Math . ceil ( ( double ) totalValueCount / config . miniBlockSizeInValues ) ; //+ 1 because first value written to header is also stored in values buffer valuesBuffer = new long [ totalMiniBlockCount * config . miniBlockSizeInValues + 1 ] ; }


Input: mini block has a size of 8 * n unpack 8 value each time [CODESPLIT] private void unpackMiniBlock ( BytePackerForLong packer ) throws IOException { for ( int j = 0 ; j < config . miniBlockSizeInValues ; j += 8 ) { unpack8Values ( packer ) ; } }


Input: Will merge the metadata as if it was coming from a single file . ( for all part files written together this will always work ) If there are conflicting values an exception will be thrown [CODESPLIT] public FileMetaData merge ( ) { String createdByString = createdBy . size ( ) == 1 ? createdBy . iterator ( ) . next ( ) : createdBy . toString ( ) ; Map < String , String > mergedKeyValues = new HashMap < String , String > ( ) ; for ( Entry < String , Set < String > > entry : keyValueMetaData . entrySet ( ) ) { if ( entry . getValue ( ) . size ( ) > 1 ) { throw new RuntimeException ( "could not merge metadata: key " + entry . getKey ( ) + " has conflicting values: " + entry . getValue ( ) ) ; } mergedKeyValues . put ( entry . getKey ( ) , entry . getValue ( ) . iterator ( ) . next ( ) ) ; } return new FileMetaData ( schema , mergedKeyValues , createdByString ) ; }


Input: Helper method to convert the old representation of logical types ( OriginalType ) to new logical type . [CODESPLIT] @ InterfaceAudience . Private public static LogicalTypeAnnotation fromOriginalType ( OriginalType originalType , DecimalMetadata decimalMetadata ) { if ( originalType == null ) { return null ; } switch ( originalType ) { case UTF8 : return stringType ( ) ; case MAP : return mapType ( ) ; case DECIMAL : int scale = ( decimalMetadata == null ? 0 : decimalMetadata . getScale ( ) ) ; int precision = ( decimalMetadata == null ? 0 : decimalMetadata . getPrecision ( ) ) ; return decimalType ( scale , precision ) ; case LIST : return listType ( ) ; case DATE : return dateType ( ) ; case INTERVAL : return IntervalLogicalTypeAnnotation . getInstance ( ) ; case TIMESTAMP_MILLIS : return timestampType ( true , LogicalTypeAnnotation . TimeUnit . MILLIS ) ; case TIMESTAMP_MICROS : return timestampType ( true , LogicalTypeAnnotation . TimeUnit . MICROS ) ; case TIME_MILLIS : return timeType ( true , LogicalTypeAnnotation . TimeUnit . MILLIS ) ; case TIME_MICROS : return timeType ( true , LogicalTypeAnnotation . TimeUnit . MICROS ) ; case UINT_8 : return intType ( 8 , false ) ; case UINT_16 : return intType ( 16 , false ) ; case UINT_32 : return intType ( 32 , false ) ; case UINT_64 : return intType ( 64 , false ) ; case INT_8 : return intType ( 8 , true ) ; case INT_16 : return intType ( 16 , true ) ; case INT_32 : return intType ( 32 , true ) ; case INT_64 : return intType ( 64 , true ) ; case ENUM : return enumType ( ) ; case JSON : return jsonType ( ) ; case BSON : return bsonType ( ) ; case MAP_KEY_VALUE : return MapKeyValueTypeAnnotation . getInstance ( ) ; default : throw new RuntimeException ( "Can't convert original type to logical type, unknown original type " + originalType ) ; } }


Input: { [CODESPLIT] @ Override public OutputFormat < Void , Tuple > getOutputFormat ( ) throws IOException { return new ParquetOutputFormat < Tuple > ( new TupleToThriftWriteSupport ( className ) ) ; }


Input: getBytes will trigger flushing block buffer DO NOT write after getBytes () is called without calling reset () [CODESPLIT] @ Override public BytesInput getBytes ( ) { // The Page Header should include: blockSizeInValues, numberOfMiniBlocks, totalValueCount if ( deltaValuesToFlush != 0 ) { flushBlockBuffer ( ) ; } return BytesInput . concat ( config . toBytesInput ( ) , BytesInput . fromUnsignedVarInt ( totalValueCount ) , BytesInput . fromZigZagVarLong ( firstValue ) , BytesInput . from ( baos ) ) ; }


Input: Precondition - style validation that throws { @link IllegalArgumentException } . [CODESPLIT] public static void checkArgument ( boolean isValid , String message , Object ... args ) throws IllegalArgumentException { if ( ! isValid ) { throw new IllegalArgumentException ( String . format ( String . valueOf ( message ) , strings ( args ) ) ) ; } }


Input: Precondition - style validation that throws { @link IllegalStateException } . [CODESPLIT] public static void checkState ( boolean isValid , String message , Object ... args ) throws IllegalStateException { if ( ! isValid ) { throw new IllegalStateException ( String . format ( String . valueOf ( message ) , strings ( args ) ) ) ; } }


Input: Reads one record from in and writes it to out . Exceptions encountered during reading are treated as skippable exceptions { @link FieldIgnoredHandler } will be notified when registered . [CODESPLIT] @ Override public void readOne ( TProtocol in , TProtocol out ) throws TException { List < Action > buffer = new LinkedList < Action > ( ) ; try { boolean hasFieldsIgnored = readOneStruct ( in , buffer , thriftType ) ; if ( hasFieldsIgnored ) { notifyRecordHasFieldIgnored ( ) ; } } catch ( Exception e ) { throw new SkippableException ( error ( "Error while reading" , buffer ) , e ) ; } try { for ( Action a : buffer ) { a . write ( out ) ; } } catch ( Exception e ) { throw new TException ( error ( "Can not write record" , buffer ) , e ) ; } }


Input: In thrift enum values are written as ints this method checks if the enum index is defined . [CODESPLIT] private void checkEnum ( ThriftType expectedType , int i ) { if ( expectedType . getType ( ) == ThriftTypeID . ENUM ) { ThriftType . EnumType expectedEnumType = ( ThriftType . EnumType ) expectedType ; if ( expectedEnumType . getEnumValueById ( i ) == null ) { throw new DecodingSchemaMismatchException ( "can not find index " + i + " in enum " + expectedType ) ; } } }


Input: attempts to validate and construct a { @link MessageType } from a read projection schema [CODESPLIT] public static MessageType getSchemaForRead ( MessageType fileMessageType , String partialReadSchemaString ) { if ( partialReadSchemaString == null ) return fileMessageType ; MessageType requestedMessageType = MessageTypeParser . parseMessageType ( partialReadSchemaString ) ; return getSchemaForRead ( fileMessageType , requestedMessageType ) ; }


Input: called in { @link org . apache . hadoop . mapreduce . InputFormat#getSplits ( org . apache . hadoop . mapreduce . JobContext ) } in the front end [CODESPLIT] @ Deprecated public ReadContext init ( Configuration configuration , Map < String , String > keyValueMetaData , MessageType fileSchema ) { throw new UnsupportedOperationException ( "Override init(InitContext)" ) ; }


Input: called in { @link org . apache . hadoop . mapreduce . InputFormat#getSplits ( org . apache . hadoop . mapreduce . JobContext ) } in the front end [CODESPLIT] public ReadContext init ( InitContext context ) { return init ( context . getConfiguration ( ) , context . getMergedKeyValueMetaData ( ) , context . getFileSchema ( ) ) ; }


Input: Sets min and max values re - uses the byte [] passed in . Any changes made to byte [] will be reflected in min and max values as well . [CODESPLIT] @ Override public void setMinMaxFromBytes ( byte [ ] minBytes , byte [ ] maxBytes ) { max = Binary . fromReusedByteArray ( maxBytes ) ; min = Binary . fromReusedByteArray ( minBytes ) ; this . markAsNotEmpty ( ) ; }


Input: iterate through values in each mini block and calculate the bitWidths of max values . [CODESPLIT] private void calculateBitWidthsForDeltaBlockBuffer ( int miniBlocksToFlush ) { for ( int miniBlockIndex = 0 ; miniBlockIndex < miniBlocksToFlush ; miniBlockIndex ++ ) { int mask = 0 ; int miniStart = miniBlockIndex * config . miniBlockSizeInValues ; //The end of current mini block could be the end of current block(deltaValuesToFlush) buffer when data is not aligned to mini block int miniEnd = Math . min ( ( miniBlockIndex + 1 ) * config . miniBlockSizeInValues , deltaValuesToFlush ) ; for ( int i = miniStart ; i < miniEnd ; i ++ ) { mask |= deltaBlockBuffer [ i ] ; } bitWidths [ miniBlockIndex ] = 32 - Integer . numberOfLeadingZeros ( mask ) ; } }


Input: Returns builder for creating an and filter . [CODESPLIT] public static final UnboundRecordFilter and ( final UnboundRecordFilter filter1 , final UnboundRecordFilter filter2 ) { Preconditions . checkNotNull ( filter1 , "filter1" ) ; Preconditions . checkNotNull ( filter2 , "filter2" ) ; return new UnboundRecordFilter ( ) { @ Override public RecordFilter bind ( Iterable < ColumnReader > readers ) { return new AndRecordFilter ( filter1 . bind ( readers ) , filter2 . bind ( readers ) ) ; } } ; }


Input: { [CODESPLIT] @ Override public void writeToStringBuilder ( StringBuilder sb , String indent ) { sb . append ( "message " ) . append ( getName ( ) ) . append ( getLogicalTypeAnnotation ( ) == null ? "" : " (" + getLogicalTypeAnnotation ( ) . toString ( ) + ")" ) . append ( " {\n" ) ; membersDisplayString ( sb , "  " ) ; sb . append ( "}\n" ) ; }


Input: / * Creates a column index store which lazily reads column / offset indexes for the columns in paths . ( paths are the set of columns used for the projection ) [CODESPLIT] static ColumnIndexStore create ( ParquetFileReader reader , BlockMetaData block , Set < ColumnPath > paths ) { try { return new ColumnIndexStoreImpl ( reader , block , paths ) ; } catch ( MissingOffsetIndexException e ) { return EMPTY ; } }


Input: Returns builder for creating an and filter . [CODESPLIT] public static final UnboundRecordFilter not ( final UnboundRecordFilter filter ) { Preconditions . checkNotNull ( filter , "filter" ) ; return new UnboundRecordFilter ( ) { @ Override public RecordFilter bind ( Iterable < ColumnReader > readers ) { return new NotRecordFilter ( filter . bind ( readers ) ) ; } } ; }


Input: If the given throwable is an instance of E throw it as an E . [CODESPLIT] public static < E extends Exception > void throwIfInstance ( Throwable t , Class < E > excClass ) throws E { if ( excClass . isAssignableFrom ( t . getClass ( ) ) ) { // the throwable is already an exception, so return it throw excClass . cast ( t ) ; } }


Input: Returns the typed statistics object based on the passed type parameter [CODESPLIT] @ Deprecated public static Statistics getStatsBasedOnType ( PrimitiveTypeName type ) { switch ( type ) { case INT32 : return new IntStatistics ( ) ; case INT64 : return new LongStatistics ( ) ; case FLOAT : return new FloatStatistics ( ) ; case DOUBLE : return new DoubleStatistics ( ) ; case BOOLEAN : return new BooleanStatistics ( ) ; case BINARY : return new BinaryStatistics ( ) ; case INT96 : return new BinaryStatistics ( ) ; case FIXED_LEN_BYTE_ARRAY : return new BinaryStatistics ( ) ; default : throw new UnknownColumnTypeException ( type ) ; } }


Input: Creates an empty { @code Statistics } instance for the specified type to be used for reading / writing the new min / max statistics used in the V2 format . [CODESPLIT] public static Statistics < ? > createStats ( Type type ) { PrimitiveType primitive = type . asPrimitiveType ( ) ; switch ( primitive . getPrimitiveTypeName ( ) ) { case INT32 : return new IntStatistics ( primitive ) ; case INT64 : return new LongStatistics ( primitive ) ; case FLOAT : return new FloatStatistics ( primitive ) ; case DOUBLE : return new DoubleStatistics ( primitive ) ; case BOOLEAN : return new BooleanStatistics ( primitive ) ; case BINARY : case INT96 : case FIXED_LEN_BYTE_ARRAY : return new BinaryStatistics ( primitive ) ; default : throw new UnknownColumnTypeException ( primitive . getPrimitiveTypeName ( ) ) ; } }


Input: Returns a builder to create new statistics object . Used to read the statistics from the parquet file . [CODESPLIT] public static Builder getBuilderForReading ( PrimitiveType type ) { switch ( type . getPrimitiveTypeName ( ) ) { case FLOAT : return new FloatBuilder ( type ) ; case DOUBLE : return new DoubleBuilder ( type ) ; default : return new Builder ( type ) ; } }


Input: Method to merge this statistics object with the object passed as parameter . Merging keeps the smallest of min values largest of max values and combines the number of null counts . [CODESPLIT] public void mergeStatistics ( Statistics stats ) { if ( stats . isEmpty ( ) ) return ; // Merge stats only if they have the same type if ( type . equals ( stats . type ) ) { incrementNumNulls ( stats . getNumNulls ( ) ) ; if ( stats . hasNonNullValue ( ) ) { mergeStatisticsMinMax ( stats ) ; markAsNotEmpty ( ) ; } } else { throw StatisticsClassException . create ( this , stats ) ; } }


Input: Given a schema check to see if it is a union of a null type and a regular schema and then return the non - null sub - schema . Otherwise return the given schema . [CODESPLIT] public static Schema getNonNull ( Schema schema ) { if ( schema . getType ( ) . equals ( Schema . Type . UNION ) ) { List < Schema > schemas = schema . getTypes ( ) ; if ( schemas . size ( ) == 2 ) { if ( schemas . get ( 0 ) . getType ( ) . equals ( Schema . Type . NULL ) ) { return schemas . get ( 1 ) ; } else if ( schemas . get ( 1 ) . getType ( ) . equals ( Schema . Type . NULL ) ) { return schemas . get ( 0 ) ; } else { return schema ; } } else { return schema ; } } else { return schema ; } }


Input: Implements the rules for interpreting existing data from the logical type spec for the LIST annotation . This is used to produce the expected schema . <p > The AvroArrayConverter will decide whether the repeated type is the array element type by testing whether the element schema and repeated type are the same . This ensures that the LIST rules are followed when there is no schema and that a schema can be provided to override the default behavior . [CODESPLIT] private boolean isElementType ( Type repeatedType , String parentName ) { return ( // can't be a synthetic layer because it would be invalid repeatedType . isPrimitive ( ) || repeatedType . asGroupType ( ) . getFieldCount ( ) > 1 || repeatedType . asGroupType ( ) . getType ( 0 ) . isRepetition ( REPEATED ) || // known patterns without the synthetic layer repeatedType . getName ( ) . equals ( "array" ) || repeatedType . getName ( ) . equals ( parentName + "_tuple" ) || // default assumption assumeRepeatedIsListElement ) ; }


Input: Creates JobContext from a JobConf and jobId using the correct constructor for based on Hadoop version . <code > jobId< / code > could be null . [CODESPLIT] public static JobContext newJobContext ( Configuration conf , JobID jobId ) { try { return ( JobContext ) JOB_CONTEXT_CONSTRUCTOR . newInstance ( conf , jobId ) ; } catch ( InstantiationException e ) { throw new IllegalArgumentException ( "Can't instantiate JobContext" , e ) ; } catch ( IllegalAccessException e ) { throw new IllegalArgumentException ( "Can't instantiate JobContext" , e ) ; } catch ( InvocationTargetException e ) { throw new IllegalArgumentException ( "Can't instantiate JobContext" , e ) ; } }


Input: Creates TaskAttemptContext from a JobConf and jobId using the correct constructor for based on Hadoop version . [CODESPLIT] public static TaskAttemptContext newTaskAttemptContext ( Configuration conf , TaskAttemptID taskAttemptId ) { try { return ( TaskAttemptContext ) TASK_CONTEXT_CONSTRUCTOR . newInstance ( conf , taskAttemptId ) ; } catch ( InstantiationException e ) { throw new IllegalArgumentException ( "Can't instantiate TaskAttemptContext" , e ) ; } catch ( IllegalAccessException e ) { throw new IllegalArgumentException ( "Can't instantiate TaskAttemptContext" , e ) ; } catch ( InvocationTargetException e ) { throw new IllegalArgumentException ( "Can't instantiate TaskAttemptContext" , e ) ; } }


Input: Invoke getConfiguration () method on JobContext . Works with both Hadoop 1 and 2 . [CODESPLIT] public static Configuration getConfiguration ( JobContext context ) { try { return ( Configuration ) GET_CONFIGURATION_METHOD . invoke ( context ) ; } catch ( IllegalAccessException e ) { throw new IllegalArgumentException ( "Can't invoke method" , e ) ; } catch ( InvocationTargetException e ) { throw new IllegalArgumentException ( "Can't invoke method" , e ) ; } }


Input: Invokes a method and rethrows any exception as runtime exceptions . [CODESPLIT] private static Object invoke ( Method method , Object obj , Object ... args ) { try { return method . invoke ( obj , args ) ; } catch ( IllegalAccessException e ) { throw new IllegalArgumentException ( "Can't invoke method " + method . getName ( ) , e ) ; } catch ( InvocationTargetException e ) { throw new IllegalArgumentException ( "Can't invoke method " + method . getName ( ) , e ) ; } }


Input: appends a display string for of the members of this group to sb [CODESPLIT] void membersDisplayString ( StringBuilder sb , String indent ) { for ( Type field : fields ) { field . writeToStringBuilder ( sb , indent ) ; if ( field . isPrimitive ( ) ) { sb . append ( ";" ) ; } sb . append ( "\n" ) ; } }


Input: { [CODESPLIT] @ Override public void writeToStringBuilder ( StringBuilder sb , String indent ) { sb . append ( indent ) . append ( getRepetition ( ) . name ( ) . toLowerCase ( Locale . ENGLISH ) ) . append ( " group " ) . append ( getName ( ) ) . append ( getLogicalTypeAnnotation ( ) == null ? "" : " (" + getLogicalTypeAnnotation ( ) . toString ( ) + ")" ) . append ( getId ( ) == null ? "" : " = " + getId ( ) ) . append ( " {\n" ) ; membersDisplayString ( sb , indent + "  " ) ; sb . append ( indent ) . append ( "}" ) ; }


Input: produces the list of fields resulting from merging toMerge into the fields of this [CODESPLIT] List < Type > mergeFields ( GroupType toMerge , boolean strict ) { List < Type > newFields = new ArrayList < Type > ( ) ; // merge existing fields for ( Type type : this . getFields ( ) ) { Type merged ; if ( toMerge . containsField ( type . getName ( ) ) ) { Type fieldToMerge = toMerge . getType ( type . getName ( ) ) ; if ( type . getLogicalTypeAnnotation ( ) != null && ! type . getLogicalTypeAnnotation ( ) . equals ( fieldToMerge . getLogicalTypeAnnotation ( ) ) ) { throw new IncompatibleSchemaModificationException ( "cannot merge logical type " + fieldToMerge . getLogicalTypeAnnotation ( ) + " into " + type . getLogicalTypeAnnotation ( ) ) ; } merged = type . union ( fieldToMerge , strict ) ; } else { merged = type ; } newFields . add ( merged ) ; } // add new fields for ( Type type : toMerge . getFields ( ) ) { if ( ! this . containsField ( type . getName ( ) ) ) { newFields . add ( type ) ; } } return newFields ; }


Input: Init counters in hadoop s mapred API which is used by cascading and Hive . [CODESPLIT] public static void initCounterFromReporter ( Reporter reporter , Configuration configuration ) { counterLoader = new MapRedCounterLoader ( reporter , configuration ) ; loadCounters ( ) ; }


Input: Fills specified buffer with compressed data . Returns actual number of bytes of compressed data . A return value of 0 indicates that needsInput () should be called in order to determine if more input data is required . [CODESPLIT] @ Override public synchronized int compress ( byte [ ] buffer , int off , int len ) throws IOException { SnappyUtil . validateBuffer ( buffer , off , len ) ; if ( needsInput ( ) ) { // No buffered output bytes and no input to consume, need more input return 0 ; } if ( ! outputBuffer . hasRemaining ( ) ) { // There is uncompressed input, compress it now int maxOutputSize = Snappy . maxCompressedLength ( inputBuffer . position ( ) ) ; if ( maxOutputSize > outputBuffer . capacity ( ) ) { ByteBuffer oldBuffer = outputBuffer ; outputBuffer = ByteBuffer . allocateDirect ( maxOutputSize ) ; CleanUtil . clean ( oldBuffer ) ; } // Reset the previous outputBuffer outputBuffer . clear ( ) ; inputBuffer . limit ( inputBuffer . position ( ) ) ; inputBuffer . position ( 0 ) ; int size = Snappy . compress ( inputBuffer , outputBuffer ) ; outputBuffer . limit ( size ) ; inputBuffer . limit ( 0 ) ; inputBuffer . rewind ( ) ; } // Return compressed output up to 'len' int numBytes = Math . min ( len , outputBuffer . remaining ( ) ) ; outputBuffer . get ( buffer , off , numBytes ) ; bytesWritten += numBytes ; return numBytes ; }


Input: Closes a ( potentially null ) closeable swallowing any IOExceptions thrown by c . close () . The exception will be logged . [CODESPLIT] public static void closeAndSwallowIOExceptions ( Closeable c ) { if ( c == null ) { return ; } try { c . close ( ) ; } catch ( IOException e ) { LOG . warn ( "Encountered exception closing closeable" , e ) ; } }


Input: Returns a non - null Filter which is a wrapper around either a FilterPredicate an UnboundRecordFilter or a no - op filter . [CODESPLIT] public static Filter getFilter ( Configuration conf ) { return FilterCompat . get ( getFilterPredicate ( conf ) , getUnboundRecordFilterInstance ( conf ) ) ; }


Input: { [CODESPLIT] @ Override public RecordReader < Void , T > createRecordReader ( InputSplit inputSplit , TaskAttemptContext taskAttemptContext ) throws IOException , InterruptedException { Configuration conf = ContextUtil . getConfiguration ( taskAttemptContext ) ; ReadSupport < T > readSupport = getReadSupport ( conf ) ; return new ParquetRecordReader < T > ( readSupport , getFilter ( conf ) ) ; }


Input: { [CODESPLIT] @ Override public List < InputSplit > getSplits ( JobContext jobContext ) throws IOException { Configuration configuration = ContextUtil . getConfiguration ( jobContext ) ; List < InputSplit > splits = new ArrayList < InputSplit > ( ) ; if ( isTaskSideMetaData ( configuration ) ) { // Although not required by the API, some clients may depend on always // receiving ParquetInputSplit. Translation is required at some point. for ( InputSplit split : super . getSplits ( jobContext ) ) { Preconditions . checkArgument ( split instanceof FileSplit , "Cannot wrap non-FileSplit: " + split ) ; splits . add ( ParquetInputSplit . from ( ( FileSplit ) split ) ) ; } return splits ; } else { splits . addAll ( getSplits ( configuration , getFooters ( jobContext ) ) ) ; } return splits ; }


Input: / * This is to support multi - level / recursive directory listing until MAPREDUCE - 1577 is fixed . [CODESPLIT] @ Override protected List < FileStatus > listStatus ( JobContext jobContext ) throws IOException { return getAllFileRecursively ( super . listStatus ( jobContext ) , ContextUtil . getConfiguration ( jobContext ) ) ; }


Input: the footers for the files [CODESPLIT] public List < Footer > getFooters ( Configuration configuration , Collection < FileStatus > statuses ) throws IOException { LOG . debug ( "reading {} files" , statuses . size ( ) ) ; boolean taskSideMetaData = isTaskSideMetaData ( configuration ) ; return ParquetFileReader . readAllFootersInParallelUsingSummaryFiles ( configuration , statuses , taskSideMetaData ) ; }


Input: groups together all the data blocks for the same HDFS block [CODESPLIT] static < T > List < ParquetInputSplit > generateSplits ( List < BlockMetaData > rowGroupBlocks , BlockLocation [ ] hdfsBlocksArray , FileStatus fileStatus , String requestedSchema , Map < String , String > readSupportMetadata , long minSplitSize , long maxSplitSize ) throws IOException { List < SplitInfo > splitRowGroups = generateSplitInfo ( rowGroupBlocks , hdfsBlocksArray , minSplitSize , maxSplitSize ) ; //generate splits from rowGroups of each split List < ParquetInputSplit > resultSplits = new ArrayList < ParquetInputSplit > ( ) ; for ( SplitInfo splitInfo : splitRowGroups ) { ParquetInputSplit split = splitInfo . getParquetInputSplit ( fileStatus , requestedSchema , readSupportMetadata ) ; resultSplits . add ( split ) ; } return resultSplits ; }


Input: Keeps records if their value is equal to the provided value . Nulls are treated the same way the java programming language does . <p > For example : eq ( column null ) will keep all records whose value is null . eq ( column 7 ) will keep all records whose value is 7 and will drop records whose value is null [CODESPLIT] public static < T extends Comparable < T > , C extends Column < T > & SupportsEqNotEq > Eq < T > eq ( C column , T value ) { return new Eq < T > ( column , value ) ; }


Input: Keeps records if their value is not equal to the provided value . Nulls are treated the same way the java programming language does . <p > For example : notEq ( column null ) will keep all records whose value is not null . notEq ( column 7 ) will keep all records whose value is not 7 including records whose value is null . [CODESPLIT] public static < T extends Comparable < T > , C extends Column < T > & SupportsEqNotEq > NotEq < T > notEq ( C column , T value ) { return new NotEq < T > ( column , value ) ; }


Input: Keeps records if their value is less than ( but not equal to ) the provided value . The provided value cannot be null as less than null has no meaning . Records with null values will be dropped . <p > For example : lt ( column 7 ) will keep all records whose value is less than ( but not equal to ) 7 and not null . [CODESPLIT] public static < T extends Comparable < T > , C extends Column < T > & SupportsLtGt > Lt < T > lt ( C column , T value ) { return new Lt < T > ( column , value ) ; }


Input: Keeps records if their value is less than or equal to the provided value . The provided value cannot be null as less than null has no meaning . Records with null values will be dropped . <p > For example : ltEq ( column 7 ) will keep all records whose value is less than or equal to 7 and not null . [CODESPLIT] public static < T extends Comparable < T > , C extends Column < T > & SupportsLtGt > LtEq < T > ltEq ( C column , T value ) { return new LtEq < T > ( column , value ) ; }


Input: Keeps records if their value is greater than ( but not equal to ) the provided value . The provided value cannot be null as less than null has no meaning . Records with null values will be dropped . <p > For example : gt ( column 7 ) will keep all records whose value is greater than ( but not equal to ) 7 and not null . [CODESPLIT] public static < T extends Comparable < T > , C extends Column < T > & SupportsLtGt > Gt < T > gt ( C column , T value ) { return new Gt < T > ( column , value ) ; }


Input: Keeps records if their value is greater than or equal to the provided value . The provided value cannot be null as less than null has no meaning . Records with null values will be dropped . <p > For example : gtEq ( column 7 ) will keep all records whose value is greater than or equal to 7 and not null . [CODESPLIT] public static < T extends Comparable < T > , C extends Column < T > & SupportsLtGt > GtEq < T > gtEq ( C column , T value ) { return new GtEq < T > ( column , value ) ; }


Input: Keeps records that pass the provided { @link UserDefinedPredicate } <p > The provided class must have a default constructor . To use an instance of a UserDefinedPredicate instead see userDefined below . [CODESPLIT] public static < T extends Comparable < T > , U extends UserDefinedPredicate < T > > UserDefined < T , U > userDefined ( Column < T > column , Class < U > clazz ) { return new UserDefinedByClass < T , U > ( column , clazz ) ; }


Input: Keeps records that pass the provided { @link UserDefinedPredicate } <p > The provided instance of UserDefinedPredicate must be serializable . [CODESPLIT] public static < T extends Comparable < T > , U extends UserDefinedPredicate < T > & Serializable > UserDefined < T , U > userDefined ( Column < T > column , U udp ) { return new UserDefinedByInstance < T , U > ( column , udp ) ; }


Input: prints a debug message [CODESPLIT] public void debug ( Object m ) { if ( m instanceof Throwable ) { logger . debug ( "" , ( Throwable ) m ) ; } else { logger . debug ( String . valueOf ( m ) ) ; } }


Input: prints a debug message [CODESPLIT] public void debug ( Object m , Throwable t ) { logger . debug ( String . valueOf ( m ) , t ) ; }


Input: prints an info message [CODESPLIT] public void info ( Object m ) { if ( m instanceof Throwable ) { logger . info ( "" , ( Throwable ) m ) ; } else { logger . info ( String . valueOf ( m ) ) ; } }


Input: prints an info message [CODESPLIT] public void info ( Object m , Throwable t ) { logger . info ( String . valueOf ( m ) , t ) ; }


Input: prints a warn message [CODESPLIT] public void warn ( Object m ) { if ( m instanceof Throwable ) { logger . warn ( "" , ( Throwable ) m ) ; } else { logger . warn ( String . valueOf ( m ) ) ; } }


Input: prints a warn message [CODESPLIT] public void warn ( Object m , Throwable t ) { logger . warn ( String . valueOf ( m ) , t ) ; }


Input: prints an error message [CODESPLIT] public void error ( Object m ) { if ( m instanceof Throwable ) { logger . error ( "" , ( Throwable ) m ) ; } else { logger . error ( String . valueOf ( m ) ) ; } }


Input: prints an error message [CODESPLIT] public void error ( Object m , Throwable t ) { logger . error ( String . valueOf ( m ) , t ) ; }


Input: Returns builder for creating an and filter . [CODESPLIT] public static final UnboundRecordFilter or ( final UnboundRecordFilter filter1 , final UnboundRecordFilter filter2 ) { Preconditions . checkNotNull ( filter1 , "filter1" ) ; Preconditions . checkNotNull ( filter2 , "filter2" ) ; return new UnboundRecordFilter ( ) { @ Override public RecordFilter bind ( Iterable < ColumnReader > readers ) { return new OrRecordFilter ( filter1 . bind ( readers ) , filter2 . bind ( readers ) ) ; } } ; }


Input: Writes Protocol buffer to parquet file . [CODESPLIT] @ Override public void write ( T record ) { recordConsumer . startMessage ( ) ; try { messageWriter . writeTopLevelMessage ( record ) ; } catch ( RuntimeException e ) { Message m = ( record instanceof Message . Builder ) ? ( ( Message . Builder ) record ) . build ( ) : ( Message ) record ; LOG . error ( "Cannot write message " + e . getMessage ( ) + " : " + m ) ; throw e ; } recordConsumer . endMessage ( ) ; }


Input: validates mapping between protobuffer fields and parquet fields . [CODESPLIT] private void validatedMapping ( Descriptor descriptor , GroupType parquetSchema ) { List < FieldDescriptor > allFields = descriptor . getFields ( ) ; for ( FieldDescriptor fieldDescriptor : allFields ) { String fieldName = fieldDescriptor . getName ( ) ; int fieldIndex = fieldDescriptor . getIndex ( ) ; int parquetIndex = parquetSchema . getFieldIndex ( fieldName ) ; if ( fieldIndex != parquetIndex ) { String message = "FieldIndex mismatch name=" + fieldName + ": " + fieldIndex + " != " + parquetIndex ; throw new IncompatibleSchemaModificationException ( message ) ; } } }


Input: Returns message descriptor as JSON String [CODESPLIT] private String serializeDescriptor ( Class < ? extends Message > protoClass ) { Descriptor descriptor = Protobufs . getMessageDescriptor ( protoClass ) ; DescriptorProtos . DescriptorProto asProto = descriptor . toProto ( ) ; return TextFormat . printToString ( asProto ) ; }


Input: Create a codec factory that will provide compressors and decompressors that will work natively with ByteBuffers backed by direct memory . [CODESPLIT] public static CodecFactory createDirectCodecFactory ( Configuration config , ByteBufferAllocator allocator , int pageSize ) { return new DirectCodecFactory ( config , allocator , pageSize ) ; }


Input: a . 2 . b [ 3 ] [ key ] * optional ( union with null ) should be ignored * unions should match by position number or short name ( e . g . 2 user ) * fields should match by name * arrays are dereferenced by position [ n ] = &gt ; schema is the element schema * maps are dereferenced by key = &gt ; schema is the value schema [CODESPLIT] public static Schema filterSchema ( Schema schema , String ... fieldPaths ) { return filterSchema ( schema , Lists . newArrayList ( fieldPaths ) ) ; }


Input: Given a thrift definition protocols events it checks all the required fields and create default value if a required field is missing [CODESPLIT] public List < TProtocol > amendMissingRequiredFields ( StructType recordThriftType ) throws TException { Iterator < TProtocol > protocolIter = rootEvents . iterator ( ) ; checkStruct ( protocolIter , recordThriftType ) ; return fixedEvents ; }


Input: check each element of the Set make sure all the element contain required fields [CODESPLIT] private void checkSet ( Iterator < TProtocol > eventIter , ThriftField setFieldDefinition ) throws TException { TSet thriftSet = acceptProtocol ( eventIter . next ( ) ) . readSetBegin ( ) ; ThriftField elementFieldDefinition = ( ( ThriftType . SetType ) setFieldDefinition . getType ( ) ) . getValues ( ) ; int setSize = thriftSet . size ; for ( int i = 0 ; i < setSize ; i ++ ) { checkField ( thriftSet . elemType , eventIter , elementFieldDefinition ) ; } acceptProtocol ( eventIter . next ( ) ) . readSetEnd ( ) ; }


Input: reads the meta data from the stream [CODESPLIT] public static FileMetaData readFileMetaData ( InputStream from , boolean skipRowGroups ) throws IOException { FileMetaData md = new FileMetaData ( ) ; if ( skipRowGroups ) { readFileMetaData ( from , new DefaultFileMetaDataConsumer ( md ) , skipRowGroups ) ; } else { read ( from , md ) ; } return md ; }


Input: close the file [CODESPLIT] @ Override public void close ( ) throws IOException { try { recordWriter . close ( taskAttemptContext ) ; } catch ( InterruptedException e ) { Thread . interrupted ( ) ; throw new IOException ( "The thread was interrupted" , e ) ; } }


Input: set the schema being written to the job conf [CODESPLIT] public static void setSchema ( Job job , MessageType schema ) { GroupWriteSupport . setSchema ( schema , ContextUtil . getConfiguration ( job ) ) ; }


Input: set up the mapping in both directions [CODESPLIT] private static void add ( Class < ? > c , PrimitiveTypeName p ) { Set < PrimitiveTypeName > descriptors = classToParquetType . get ( c ) ; if ( descriptors == null ) { descriptors = new HashSet < PrimitiveTypeName > ( ) ; classToParquetType . put ( c , descriptors ) ; } descriptors . add ( p ) ; Set < Class < ? > > classes = parquetTypeToClass . get ( p ) ; if ( classes == null ) { classes = new HashSet < Class < ? > > ( ) ; parquetTypeToClass . put ( p , classes ) ; } classes . add ( c ) ; }