Input: Asserts that foundColumn was declared as a type that is compatible with the type for this column found in the schema of the parquet file . [CODESPLIT] public static < T extends Comparable < T > > void assertTypeValid ( Column < T > foundColumn , PrimitiveTypeName primitiveType ) { Class < T > foundColumnType = foundColumn . getColumnType ( ) ; ColumnPath columnPath = foundColumn . getColumnPath ( ) ; Set < PrimitiveTypeName > validTypeDescriptors = classToParquetType . get ( foundColumnType ) ; if ( validTypeDescriptors == null ) { StringBuilder message = new StringBuilder ( ) ; message . append ( "Column " ) . append ( columnPath . toDotString ( ) ) . append ( " was declared as type: " ) . append ( foundColumnType . getName ( ) ) . append ( " which is not supported in FilterPredicates." ) ; Set < Class < ? > > supportedTypes = parquetTypeToClass . get ( primitiveType ) ; if ( supportedTypes != null ) { message . append ( " Supported types for this column are: " ) . append ( supportedTypes ) ; } else { message . append ( " There are no supported types for columns of " + primitiveType ) ; } throw new IllegalArgumentException ( message . toString ( ) ) ; } if ( ! validTypeDescriptors . contains ( primitiveType ) ) { StringBuilder message = new StringBuilder ( ) ; message . append ( "FilterPredicate column: " ) . append ( columnPath . toDotString ( ) ) . append ( "'s declared type (" ) . append ( foundColumnType . getName ( ) ) . append ( ") does not match the schema found in file metadata. Column " ) . append ( columnPath . toDotString ( ) ) . append ( " is of type: " ) . append ( primitiveType ) . append ( "\nValid types for this column are: " ) . append ( parquetTypeToClass . get ( primitiveType ) ) ; throw new IllegalArgumentException ( message . toString ( ) ) ; } }


Input: Join an Iterable of Strings into a single string with a delimiter . For example join ( Arrays . asList ( foo bar x ) | ) would return foo||bar|x [CODESPLIT] public static String join ( Iterable < String > s , String on ) { return join ( s . iterator ( ) , on ) ; }


Input: Join an Iterator of Strings into a single string with a delimiter . For example join ( Arrays . asList ( foo bar x ) | ) would return foo||bar|x [CODESPLIT] public static String join ( Iterator < String > iter , String on ) { StringBuilder sb = new StringBuilder ( ) ; while ( iter . hasNext ( ) ) { sb . append ( iter . next ( ) ) ; if ( iter . hasNext ( ) ) { sb . append ( on ) ; } } return sb . toString ( ) ; }


Input: Join an Array of Strings into a single string with a delimiter . For example join ( new String [] { foo bar x } | ) would return foo||bar|x [CODESPLIT] public static String join ( String [ ] s , String on ) { return join ( Arrays . asList ( s ) , on ) ; }


Input: Expands a string according to { @link #expandGlob ( String ) } and then constructs a { @link WildcardPath } for each expanded result which can be used to match strings as described in { @link WildcardPath } . [CODESPLIT] public static List < WildcardPath > expandGlobToWildCardPaths ( String globPattern , char delim ) { List < WildcardPath > ret = new ArrayList < WildcardPath > ( ) ; for ( String expandedGlob : Strings . expandGlob ( globPattern ) ) { ret . add ( new WildcardPath ( globPattern , expandedGlob , delim ) ) ; } return ret ; }


Input: Cast value to a an int or throw an exception if there is an overflow . [CODESPLIT] public static int checkedCast ( long value ) { int valueI = ( int ) value ; if ( valueI != value ) { throw new IllegalArgumentException ( String . format ( "Overflow casting %d to an int" , value ) ) ; } return valueI ; }


Input: Internal method visible for testing purposes [CODESPLIT] @ SuppressWarnings ( "rawtypes" ) Class < ? extends HiveBinding > create ( ClassLoader classLoader ) { // HiveVersionInfo was added in 0.11, if the class does // not exist then return the hive binding for 0.10 Class hiveVersionInfo ; try { hiveVersionInfo = Class . forName ( HIVE_VERSION_CLASS_NAME , true , classLoader ) ; } catch ( ClassNotFoundException e ) { LOG . debug ( "Class " + HIVE_VERSION_CLASS_NAME + ", not found, returning {}" , Hive010Binding . class . getSimpleName ( ) ) ; return Hive010Binding . class ; } return createInternal ( hiveVersionInfo ) ; }


Input: Internal method visible for testing purposes [CODESPLIT] @ SuppressWarnings ( { "unchecked" , "rawtypes" } ) Class < ? extends HiveBinding > createInternal ( Class hiveVersionInfo ) { String hiveVersion ; try { Method getVersionMethod = hiveVersionInfo . getMethod ( HIVE_VERSION_METHOD_NAME , ( Class [ ] ) null ) ; String rawVersion = ( String ) getVersionMethod . invoke ( null , ( Object [ ] ) null ) ; LOG . debug ( "Raw Version from {} is '{}'" , hiveVersionInfo . getSimpleName ( ) , rawVersion ) ; hiveVersion = trimVersion ( rawVersion ) ; } catch ( Exception e ) { throw new UnexpectedHiveVersionProviderError ( "Unexpected error whilst " + "determining Hive version" , e ) ; } if ( hiveVersion . equalsIgnoreCase ( HIVE_VERSION_UNKNOWN ) ) { LOG . debug ( "Unknown hive version, attempting to guess" ) ; return createBindingForUnknownVersion ( ) ; } if ( hiveVersion . startsWith ( HIVE_VERSION_010 ) ) { LOG . debug ( "Hive version {}, returning {}" , hiveVersion , Hive010Binding . class . getSimpleName ( ) ) ; return Hive010Binding . class ; } else if ( hiveVersion . startsWith ( HIVE_VERSION_011 ) ) { LOG . debug ( "Hive version " + hiveVersion + ", returning " + Hive010Binding . class . getSimpleName ( ) + " as it's expected the 0.10 " + "binding will work with 0.11" ) ; return Hive010Binding . class ; } else if ( hiveVersion . startsWith ( HIVE_VERSION_013 ) ) { throw new HiveBindingInstantiationError ( "Hive 0.13 contains native Parquet support " + "and the parquet-hive jars from the parquet project should not be included " + "in Hive's classpath." ) ; } LOG . debug ( "Hive version {}, returning {}" , hiveVersion , Hive012Binding . class . getSimpleName ( ) ) ; // as of 11/26/2013 it looks like the 0.12 binding will work for 0.13 return Hive012Binding . class ; }


Input: TODO : all the catching of Exceptions below -- see PARQUET - 383 [CODESPLIT] public static void writeMetaDataFile ( Configuration configuration , Path outputPath ) { JobSummaryLevel level = ParquetOutputFormat . getJobSummaryLevel ( configuration ) ; if ( level == JobSummaryLevel . NONE ) { return ; } try { final FileSystem fileSystem = outputPath . getFileSystem ( configuration ) ; FileStatus outputStatus = fileSystem . getFileStatus ( outputPath ) ; List < Footer > footers ; switch ( level ) { case ALL : footers = ParquetFileReader . readAllFootersInParallel ( configuration , outputStatus , false ) ; // don't skip row groups break ; case COMMON_ONLY : footers = ParquetFileReader . readAllFootersInParallel ( configuration , outputStatus , true ) ; // skip row groups break ; default : throw new IllegalArgumentException ( "Unrecognized job summary level: " + level ) ; } // If there are no footers, _metadata file cannot be written since there is no way to determine schema! // Onus of writing any summary files lies with the caller in this case. if ( footers . isEmpty ( ) ) { return ; } try { ParquetFileWriter . writeMetadataFile ( configuration , outputPath , footers , level ) ; } catch ( Exception e ) { LOG . warn ( "could not write summary file(s) for " + outputPath , e ) ; final Path metadataPath = new Path ( outputPath , ParquetFileWriter . PARQUET_METADATA_FILE ) ; try { if ( fileSystem . exists ( metadataPath ) ) { fileSystem . delete ( metadataPath , true ) ; } } catch ( Exception e2 ) { LOG . warn ( "could not delete metadata file" + outputPath , e2 ) ; } try { final Path commonMetadataPath = new Path ( outputPath , ParquetFileWriter . PARQUET_COMMON_METADATA_FILE ) ; if ( fileSystem . exists ( commonMetadataPath ) ) { fileSystem . delete ( commonMetadataPath , true ) ; } } catch ( Exception e2 ) { LOG . warn ( "could not delete metadata file" + outputPath , e2 ) ; } } } catch ( Exception e ) { LOG . warn ( "could not write summary file for " + outputPath , e ) ; } }


Input: Compatibility API [CODESPLIT] @ Deprecated public void unpack8Values ( final byte [ ] input , final int inPos , final int [ ] output , final int outPos ) { unpack8Values ( ByteBuffer . wrap ( input ) , inPos , output , outPos ) ; }


Input: Compatibility API [CODESPLIT] @ Deprecated public void unpack32Values ( byte [ ] input , int inPos , int [ ] output , int outPos ) { unpack32Values ( ByteBuffer . wrap ( input ) , inPos , output , outPos ) ; }


Input: Creates a Parquet Schema from an Arrow one and returns the mapping [CODESPLIT] public SchemaMapping fromArrow ( Schema arrowSchema ) { List < Field > fields = arrowSchema . getFields ( ) ; List < TypeMapping > parquetFields = fromArrow ( fields ) ; MessageType parquetType = addToBuilder ( parquetFields , Types . buildMessage ( ) ) . named ( "root" ) ; return new SchemaMapping ( arrowSchema , parquetType , parquetFields ) ; }


Input: Creates an Arrow Schema from an Parquet one and returns the mapping [CODESPLIT] public SchemaMapping fromParquet ( MessageType parquetSchema ) { List < Type > fields = parquetSchema . getFields ( ) ; List < TypeMapping > mappings = fromParquet ( fields ) ; List < Field > arrowFields = fields ( mappings ) ; return new SchemaMapping ( new Schema ( arrowFields ) , parquetSchema , mappings ) ; }


Input: Maps a Parquet and Arrow Schema For now does not validate primitive type compatibility [CODESPLIT] public SchemaMapping map ( Schema arrowSchema , MessageType parquetSchema ) { List < TypeMapping > children = map ( arrowSchema . getFields ( ) , parquetSchema . getFields ( ) ) ; return new SchemaMapping ( arrowSchema , parquetSchema , children ) ; }


Input: Visible for testing [CODESPLIT] List < SchemaElement > toParquetSchema ( MessageType schema ) { List < SchemaElement > result = new ArrayList < SchemaElement > ( ) ; addToList ( result , schema ) ; return result ; }


Input: Visible for testing [CODESPLIT] Set < org . apache . parquet . column . Encoding > fromFormatEncodings ( List < Encoding > encodings ) { Set < org . apache . parquet . column . Encoding > converted = new HashSet < org . apache . parquet . column . Encoding > ( ) ; for ( Encoding encoding : encodings ) { converted . add ( getEncoding ( encoding ) ) ; } // make converted unmodifiable, drop reference to modifiable copy converted = Collections . unmodifiableSet ( converted ) ; // atomically update the cache Set < org . apache . parquet . column . Encoding > cached = cachedEncodingSets . putIfAbsent ( converted , converted ) ; if ( cached == null ) { // cached == null signifies that converted was *not* in the cache previously // so we can return converted instead of throwing it away, it has now // been cached cached = converted ; } return cached ; }


Input: Visible for testing [CODESPLIT] static org . apache . parquet . column . statistics . Statistics fromParquetStatisticsInternal ( String createdBy , Statistics formatStats , PrimitiveType type , SortOrder typeSortOrder ) { // create stats object based on the column type org . apache . parquet . column . statistics . Statistics . Builder statsBuilder = org . apache . parquet . column . statistics . Statistics . getBuilderForReading ( type ) ; if ( formatStats != null ) { // Use the new V2 min-max statistics over the former one if it is filled if ( formatStats . isSetMin_value ( ) && formatStats . isSetMax_value ( ) ) { byte [ ] min = formatStats . min_value . array ( ) ; byte [ ] max = formatStats . max_value . array ( ) ; if ( isMinMaxStatsSupported ( type ) || Arrays . equals ( min , max ) ) { statsBuilder . withMin ( min ) ; statsBuilder . withMax ( max ) ; } } else { boolean isSet = formatStats . isSetMax ( ) && formatStats . isSetMin ( ) ; boolean maxEqualsMin = isSet ? Arrays . equals ( formatStats . getMin ( ) , formatStats . getMax ( ) ) : false ; boolean sortOrdersMatch = SortOrder . SIGNED == typeSortOrder ; // NOTE: See docs in CorruptStatistics for explanation of why this check is needed // The sort order is checked to avoid returning min/max stats that are not // valid with the type's sort order. In previous releases, all stats were // aggregated using a signed byte-wise ordering, which isn't valid for all the // types (e.g. strings, decimals etc.). if ( ! CorruptStatistics . shouldIgnoreStatistics ( createdBy , type . getPrimitiveTypeName ( ) ) && ( sortOrdersMatch || maxEqualsMin ) ) { if ( isSet ) { statsBuilder . withMin ( formatStats . min . array ( ) ) ; statsBuilder . withMax ( formatStats . max . array ( ) ) ; } } } if ( formatStats . isSetNull_count ( ) ) { statsBuilder . withNumNulls ( formatStats . null_count ) ; } } return statsBuilder . build ( ) ; }


Input: Returns whether to use signed order min and max with a type . It is safe to use signed min and max when the type is a string type and contains only ASCII characters ( where the sign bit was 0 ) . This checks whether the type is a string type and uses { @code useSignedStringMinMax } to determine if only ASCII characters were written . [CODESPLIT] private boolean overrideSortOrderToSigned ( PrimitiveType type ) { // even if the override is set, only return stats for string-ish types // a null type annotation is considered string-ish because some writers // failed to use the UTF8 annotation. LogicalTypeAnnotation annotation = type . getLogicalTypeAnnotation ( ) ; return useSignedStringMinMax && PrimitiveTypeName . BINARY == type . getPrimitiveTypeName ( ) && ( annotation == null || STRING_TYPES . contains ( annotation . getClass ( ) ) ) ; }


Input: Visible for testing [CODESPLIT] Type getType ( PrimitiveTypeName type ) { switch ( type ) { case INT64 : return Type . INT64 ; case INT32 : return Type . INT32 ; case BOOLEAN : return Type . BOOLEAN ; case BINARY : return Type . BYTE_ARRAY ; case FLOAT : return Type . FLOAT ; case DOUBLE : return Type . DOUBLE ; case INT96 : return Type . INT96 ; case FIXED_LEN_BYTE_ARRAY : return Type . FIXED_LEN_BYTE_ARRAY ; default : throw new RuntimeException ( "Unknown primitive type " + type ) ; } }


Input: Visible for testing [CODESPLIT] LogicalTypeAnnotation getLogicalTypeAnnotation ( ConvertedType type , SchemaElement schemaElement ) { switch ( type ) { case UTF8 : return LogicalTypeAnnotation . stringType ( ) ; case MAP : return LogicalTypeAnnotation . mapType ( ) ; case MAP_KEY_VALUE : return LogicalTypeAnnotation . MapKeyValueTypeAnnotation . getInstance ( ) ; case LIST : return LogicalTypeAnnotation . listType ( ) ; case ENUM : return LogicalTypeAnnotation . enumType ( ) ; case DECIMAL : int scale = ( schemaElement == null ? 0 : schemaElement . scale ) ; int precision = ( schemaElement == null ? 0 : schemaElement . precision ) ; return LogicalTypeAnnotation . decimalType ( scale , precision ) ; case DATE : return LogicalTypeAnnotation . dateType ( ) ; case TIME_MILLIS : return LogicalTypeAnnotation . timeType ( true , LogicalTypeAnnotation . TimeUnit . MILLIS ) ; case TIME_MICROS : return LogicalTypeAnnotation . timeType ( true , LogicalTypeAnnotation . TimeUnit . MICROS ) ; case TIMESTAMP_MILLIS : return LogicalTypeAnnotation . timestampType ( true , LogicalTypeAnnotation . TimeUnit . MILLIS ) ; case TIMESTAMP_MICROS : return LogicalTypeAnnotation . timestampType ( true , LogicalTypeAnnotation . TimeUnit . MICROS ) ; case INTERVAL : return LogicalTypeAnnotation . IntervalLogicalTypeAnnotation . getInstance ( ) ; case INT_8 : return LogicalTypeAnnotation . intType ( 8 , true ) ; case INT_16 : return LogicalTypeAnnotation . intType ( 16 , true ) ; case INT_32 : return LogicalTypeAnnotation . intType ( 32 , true ) ; case INT_64 : return LogicalTypeAnnotation . intType ( 64 , true ) ; case UINT_8 : return LogicalTypeAnnotation . intType ( 8 , false ) ; case UINT_16 : return LogicalTypeAnnotation . intType ( 16 , false ) ; case UINT_32 : return LogicalTypeAnnotation . intType ( 32 , false ) ; case UINT_64 : return LogicalTypeAnnotation . intType ( 64 , false ) ; case JSON : return LogicalTypeAnnotation . jsonType ( ) ; case BSON : return LogicalTypeAnnotation . bsonType ( ) ; default : throw new RuntimeException ( "Can't convert converted type to logical type, unknown converted type " + type ) ; } }


Input: Visible for testing [CODESPLIT] static FileMetaData filterFileMetaDataByMidpoint ( FileMetaData metaData , RangeMetadataFilter filter ) { List < RowGroup > rowGroups = metaData . getRow_groups ( ) ; List < RowGroup > newRowGroups = new ArrayList < RowGroup > ( ) ; for ( RowGroup rowGroup : rowGroups ) { long totalSize = 0 ; long startIndex = getOffset ( rowGroup . getColumns ( ) . get ( 0 ) ) ; for ( ColumnChunk col : rowGroup . getColumns ( ) ) { totalSize += col . getMeta_data ( ) . getTotal_compressed_size ( ) ; } long midPoint = startIndex + totalSize / 2 ; if ( filter . contains ( midPoint ) ) { newRowGroups . add ( rowGroup ) ; } } metaData . setRow_groups ( newRowGroups ) ; return metaData ; }


Input: Visible for testing [CODESPLIT] static FileMetaData filterFileMetaDataByStart ( FileMetaData metaData , OffsetMetadataFilter filter ) { List < RowGroup > rowGroups = metaData . getRow_groups ( ) ; List < RowGroup > newRowGroups = new ArrayList < RowGroup > ( ) ; for ( RowGroup rowGroup : rowGroups ) { long startIndex = getOffset ( rowGroup . getColumns ( ) . get ( 0 ) ) ; if ( filter . contains ( startIndex ) ) { newRowGroups . add ( rowGroup ) ; } } metaData . setRow_groups ( newRowGroups ) ; return metaData ; }


Input: Visible for testing [CODESPLIT] static long getOffset ( ColumnChunk columnChunk ) { ColumnMetaData md = columnChunk . getMeta_data ( ) ; long offset = md . getData_page_offset ( ) ; if ( md . isSetDictionary_page_offset ( ) && offset > md . getDictionary_page_offset ( ) ) { offset = md . getDictionary_page_offset ( ) ; } return offset ; }


Input: Visible for testing [CODESPLIT] MessageType fromParquetSchema ( List < SchemaElement > schema , List < ColumnOrder > columnOrders ) { Iterator < SchemaElement > iterator = schema . iterator ( ) ; SchemaElement root = iterator . next ( ) ; Types . MessageTypeBuilder builder = Types . buildMessage ( ) ; if ( root . isSetField_id ( ) ) { builder . id ( root . field_id ) ; } buildChildren ( builder , iterator , root . getNum_children ( ) , columnOrders , 0 ) ; return builder . named ( root . name ) ; }


Input: Statistics are no longer saved in page headers [CODESPLIT] @ Deprecated public void writeDataPageV2Header ( int uncompressedSize , int compressedSize , int valueCount , int nullCount , int rowCount , org . apache . parquet . column . statistics . Statistics statistics , org . apache . parquet . column . Encoding dataEncoding , int rlByteLength , int dlByteLength , OutputStream to ) throws IOException { writePageHeader ( newDataPageV2Header ( uncompressedSize , compressedSize , valueCount , nullCount , rowCount , dataEncoding , rlByteLength , dlByteLength ) , to ) ; }


Input: Writes a <code > long< / code > to the underlying output stream as eight bytes low byte first . In no exception is thrown the counter <code > written< / code > is incremented by <code > 8< / code > . [CODESPLIT] public final void writeLong ( long v ) throws IOException { writeBuffer [ 7 ] = ( byte ) ( v >>> 56 ) ; writeBuffer [ 6 ] = ( byte ) ( v >>> 48 ) ; writeBuffer [ 5 ] = ( byte ) ( v >>> 40 ) ; writeBuffer [ 4 ] = ( byte ) ( v >>> 32 ) ; writeBuffer [ 3 ] = ( byte ) ( v >>> 24 ) ; writeBuffer [ 2 ] = ( byte ) ( v >>> 16 ) ; writeBuffer [ 1 ] = ( byte ) ( v >>> 8 ) ; writeBuffer [ 0 ] = ( byte ) ( v >>> 0 ) ; out . write ( writeBuffer , 0 , 8 ) ; }


Input: Skips forwards until the filter finds the first match . Returns false if none found . [CODESPLIT] private void skipToMatch ( ) { while ( recordsRead < recordCount && ! recordFilter . isMatch ( ) ) { State currentState = getState ( 0 ) ; do { ColumnReader columnReader = currentState . column ; // currentLevel = depth + 1 at this point // set the current value if ( columnReader . getCurrentDefinitionLevel ( ) >= currentState . maxDefinitionLevel ) { columnReader . skip ( ) ; } columnReader . consume ( ) ; // Based on repetition level work out next state to go to int nextR = currentState . maxRepetitionLevel == 0 ? 0 : columnReader . getCurrentRepetitionLevel ( ) ; currentState = currentState . getNextState ( nextR ) ; } while ( currentState != null ) ; ++ recordsRead ; } }


Input: Writes an object to a configuration . [CODESPLIT] public static void writeObjectToConfAsBase64 ( String key , Object obj , Configuration conf ) throws IOException { try ( ByteArrayOutputStream baos = new ByteArrayOutputStream ( ) ) { try ( GZIPOutputStream gos = new GZIPOutputStream ( baos ) ; ObjectOutputStream oos = new ObjectOutputStream ( gos ) ) { oos . writeObject ( obj ) ; } conf . set ( key , new String ( Base64 . encodeBase64 ( baos . toByteArray ( ) ) , StandardCharsets . UTF_8 ) ) ; } }


Input: Reads an object ( that was written using { @link #writeObjectToConfAsBase64 } ) from a configuration [CODESPLIT] @ SuppressWarnings ( "unchecked" ) public static < T > T readObjectFromConfAsBase64 ( String key , Configuration conf ) throws IOException { String b64 = conf . get ( key ) ; if ( b64 == null ) { return null ; } byte [ ] bytes = Base64 . decodeBase64 ( b64 . getBytes ( StandardCharsets . UTF_8 ) ) ; try ( ByteArrayInputStream bais = new ByteArrayInputStream ( bytes ) ; GZIPInputStream gis = new GZIPInputStream ( bais ) ; ObjectInputStream ois = new ObjectInputStream ( gis ) ) { return ( T ) ois . readObject ( ) ; } catch ( ClassNotFoundException e ) { throw new IOException ( "Could not read object from config with key " + key , e ) ; } catch ( ClassCastException e ) { throw new IOException ( "Couldn't cast object read from config with key " + key , e ) ; } }


Input: Adds the specified parameters to this builder . Used by the writers to building up { @link OffsetIndex } objects to be written to the Parquet file . [CODESPLIT] public void add ( int compressedPageSize , long rowCount ) { add ( previousOffset + previousPageSize , compressedPageSize , previousRowIndex + previousRowCount ) ; previousRowCount = rowCount ; }


Input: Adds the specified parameters to this builder . Used by the metadata converter to building up { @link OffsetIndex } objects read from the Parquet file . [CODESPLIT] public void add ( long offset , int compressedPageSize , long firstRowIndex ) { previousOffset = offset ; offsets . add ( offset ) ; previousPageSize = compressedPageSize ; compressedPageSizes . add ( compressedPageSize ) ; previousRowIndex = firstRowIndex ; firstRowIndexes . add ( firstRowIndex ) ; }


Input: Builds the offset index . Used by the writers to building up { @link OffsetIndex } objects to be written to the Parquet file . [CODESPLIT] public OffsetIndex build ( long firstPageOffset ) { if ( compressedPageSizes . isEmpty ( ) ) { return null ; } long [ ] offsets = this . offsets . toLongArray ( ) ; if ( firstPageOffset != 0 ) { for ( int i = 0 , n = offsets . length ; i < n ; ++ i ) { offsets [ i ] += firstPageOffset ; } } OffsetIndexImpl offsetIndex = new OffsetIndexImpl ( ) ; offsetIndex . offsets = offsets ; offsetIndex . compressedPageSizes = compressedPageSizes . toIntArray ( ) ; offsetIndex . firstRowIndexes = firstRowIndexes . toLongArray ( ) ; return offsetIndex ; }


Input: Removes the mapping for the specified key from this cache if present . [CODESPLIT] public V remove ( final K key ) { V oldValue = cacheMap . remove ( key ) ; if ( oldValue != null ) { LOG . debug ( "Removed cache entry for '{}'" , key ) ; } return oldValue ; }


Input: Associates the specified value with the specified key in this cache . The value is only inserted if it is not null and it is considered current . If the cache previously contained a mapping for the key the old value is replaced only if the new value is newer than the old one . [CODESPLIT] public void put ( final K key , final V newValue ) { if ( newValue == null || ! newValue . isCurrent ( key ) ) { if ( LOG . isWarnEnabled ( ) ) { LOG . warn ( "Ignoring new cache entry for '{}' because it is {}" , key , ( newValue == null ? "null" : "not current" ) ) ; } return ; } V oldValue = cacheMap . get ( key ) ; if ( oldValue != null && oldValue . isNewerThan ( newValue ) ) { if ( LOG . isWarnEnabled ( ) ) { LOG . warn ( "Ignoring new cache entry for '{}' because " + "existing cache entry is newer" , key ) ; } return ; } // no existing value or new value is newer than old value oldValue = cacheMap . put ( key , newValue ) ; if ( LOG . isDebugEnabled ( ) ) { if ( oldValue == null ) { LOG . debug ( "Added new cache entry for '{}'" , key ) ; } else { LOG . debug ( "Overwrote existing cache entry for '{}'" , key ) ; } } }


Input: Returns the value to which the specified key is mapped or null if 1 ) the value is not current or 2 ) this cache contains no mapping for the key . [CODESPLIT] public V getCurrentValue ( final K key ) { V value = cacheMap . get ( key ) ; LOG . debug ( "Value for '{}' {} in cache" , key , ( value == null ? "not " : "" ) ) ; if ( value != null && ! value . isCurrent ( key ) ) { // value is not current; remove it and return null remove ( key ) ; return null ; } return value ; }


Input: Calls an appropriate write method based on the value . Value MUST not be null . [CODESPLIT] private void writeValue ( Type type , Schema avroSchema , Object value ) { Schema nonNullAvroSchema = AvroSchemaConverter . getNonNull ( avroSchema ) ; LogicalType logicalType = nonNullAvroSchema . getLogicalType ( ) ; if ( logicalType != null ) { Conversion < ? > conversion = model . getConversionByClass ( value . getClass ( ) , logicalType ) ; writeValueWithoutConversion ( type , nonNullAvroSchema , convert ( nonNullAvroSchema , logicalType , conversion , value ) ) ; } else { writeValueWithoutConversion ( type , nonNullAvroSchema , value ) ; } }


Input: Calls an appropriate write method based on the value . Value must not be null and the schema must not be nullable . [CODESPLIT] @ SuppressWarnings ( "unchecked" ) private void writeValueWithoutConversion ( Type type , Schema avroSchema , Object value ) { switch ( avroSchema . getType ( ) ) { case BOOLEAN : recordConsumer . addBoolean ( ( Boolean ) value ) ; break ; case INT : if ( value instanceof Character ) { recordConsumer . addInteger ( ( Character ) value ) ; } else { recordConsumer . addInteger ( ( ( Number ) value ) . intValue ( ) ) ; } break ; case LONG : recordConsumer . addLong ( ( ( Number ) value ) . longValue ( ) ) ; break ; case FLOAT : recordConsumer . addFloat ( ( ( Number ) value ) . floatValue ( ) ) ; break ; case DOUBLE : recordConsumer . addDouble ( ( ( Number ) value ) . doubleValue ( ) ) ; break ; case FIXED : recordConsumer . addBinary ( Binary . fromReusedByteArray ( ( ( GenericFixed ) value ) . bytes ( ) ) ) ; break ; case BYTES : if ( value instanceof byte [ ] ) { recordConsumer . addBinary ( Binary . fromReusedByteArray ( ( byte [ ] ) value ) ) ; } else { recordConsumer . addBinary ( Binary . fromReusedByteBuffer ( ( ByteBuffer ) value ) ) ; } break ; case STRING : recordConsumer . addBinary ( fromAvroString ( value ) ) ; break ; case RECORD : writeRecord ( type . asGroupType ( ) , avroSchema , value ) ; break ; case ENUM : recordConsumer . addBinary ( Binary . fromString ( value . toString ( ) ) ) ; break ; case ARRAY : listWriter . writeList ( type . asGroupType ( ) , avroSchema , value ) ; break ; case MAP : writeMap ( type . asGroupType ( ) , avroSchema , ( Map < CharSequence , ? > ) value ) ; break ; case UNION : writeUnion ( type . asGroupType ( ) , avroSchema , value ) ; break ; }


Input: Set and compile a glob pattern [CODESPLIT] public void set ( String glob ) { StringBuilder regex = new StringBuilder ( ) ; int setOpen = 0 ; int curlyOpen = 0 ; int len = glob . length ( ) ; hasWildcard = false ; for ( int i = 0 ; i < len ; i ++ ) { char c = glob . charAt ( i ) ; switch ( c ) { case BACKSLASH : if ( ++ i >= len ) { error ( "Missing escaped character" , glob , i ) ; } regex . append ( c ) . append ( glob . charAt ( i ) ) ; continue ; case ' ' : case ' ' : case ' ' : case ' ' : case ' ' : case ' ' : // escape regex special chars that are not glob special chars regex . append ( BACKSLASH ) ; break ; case ' ' : if ( i + 1 < len && glob . charAt ( i + 1 ) == ' ' ) { regex . append ( ' ' ) ; i ++ ; break ; } regex . append ( "[^" + PATH_SEPARATOR + "]" ) ; hasWildcard = true ; break ; case ' ' : regex . append ( ' ' ) ; hasWildcard = true ; continue ; case ' ' : // start of a group regex . append ( "(?:" ) ; // non-capturing curlyOpen ++ ; hasWildcard = true ; continue ; case ' ' : regex . append ( curlyOpen > 0 ? ' ' : c ) ; continue ; case ' ' : if ( curlyOpen > 0 ) { // end of a group curlyOpen -- ; regex . append ( ")" ) ; continue ; } break ; case ' ' : if ( setOpen > 0 ) { error ( "Unclosed character class" , glob , i ) ; } setOpen ++ ; hasWildcard = true ; break ; case ' ' : // ^ inside [...] can be unescaped if ( setOpen == 0 ) { regex . append ( BACKSLASH ) ; } break ; case ' ' : // [! needs to be translated to [^ regex . append ( setOpen > 0 && ' ' == glob . charAt ( i - 1 ) ? ' ' : ' ' ) ; continue ; case ' ' : // Many set errors like [][] could not be easily detected here, // as []], []-] and [-] are all valid POSIX glob and java regex. // We'll just let the regex compiler do the real work. setOpen = 0 ; break ; default : } regex . append ( c ) ; } if ( setOpen > 0 ) { error ( "Unclosed character class" , glob , len ) ; } if ( curlyOpen > 0 ) { error ( "Unclosed group" , glob , len ) ; } compiled = Pattern . compile ( regex . toString ( ) ) ; }


Input: Output content to the console or a file . [CODESPLIT] public void output ( String content , Logger console , String filename ) throws IOException { if ( filename == null || "-" . equals ( filename ) ) { console . info ( content ) ; } else { FSDataOutputStream outgoing = create ( filename ) ; try { outgoing . write ( content . getBytes ( StandardCharsets . UTF_8 ) ) ; } finally { outgoing . close ( ) ; } } }


Input: Returns a qualified { @link Path } for the { @code filename } . [CODESPLIT] public Path qualifiedPath ( String filename ) throws IOException { Path cwd = defaultFS ( ) . makeQualified ( new Path ( "." ) ) ; return new Path ( filename ) . makeQualified ( defaultFS ( ) . getUri ( ) , cwd ) ; }


Input: Returns a { @link URI } for the { @code filename } that is a qualified Path or a resource URI . [CODESPLIT] public URI qualifiedURI ( String filename ) throws IOException { URI fileURI = URI . create ( filename ) ; if ( RESOURCE_URI_SCHEME . equals ( fileURI . getScheme ( ) ) ) { return fileURI ; } else { return qualifiedPath ( filename ) . toUri ( ) ; } }


Input: Opens an existing file or resource . [CODESPLIT] public InputStream open ( String filename ) throws IOException { if ( STDIN_AS_SOURCE . equals ( filename ) ) { return System . in ; } URI uri = qualifiedURI ( filename ) ; if ( RESOURCE_URI_SCHEME . equals ( uri . getScheme ( ) ) ) { return Resources . getResource ( uri . getRawSchemeSpecificPart ( ) ) . openStream ( ) ; } else { Path filePath = new Path ( uri ) ; // even though it was qualified using the default FS, it may not be in it FileSystem fs = filePath . getFileSystem ( getConf ( ) ) ; return fs . open ( filePath ) ; } }


Input: Returns a { @link ClassLoader } for a set of jars and directories . [CODESPLIT] protected static ClassLoader loaderFor ( List < String > jars , List < String > paths ) throws MalformedURLException { return AccessController . doPrivileged ( new GetClassLoader ( urls ( jars , paths ) ) ) ; }


Input: Returns a { @link ClassLoader } for a set of jars . [CODESPLIT] protected static ClassLoader loaderForJars ( List < String > jars ) throws MalformedURLException { return AccessController . doPrivileged ( new GetClassLoader ( urls ( jars , null ) ) ) ; }


Input: Returns a { @link ClassLoader } for a set of directories . [CODESPLIT] protected static ClassLoader loaderForPaths ( List < String > paths ) throws MalformedURLException { return AccessController . doPrivileged ( new GetClassLoader ( urls ( null , paths ) ) ) ; }


Input: { [CODESPLIT] @ Override public void writeInteger ( int v ) { try { bitPackingWriter . write ( v ) ; } catch ( IOException e ) { throw new ParquetEncodingException ( e ) ; } }


Input: { [CODESPLIT] @ Override public BytesInput getBytes ( ) { try { this . bitPackingWriter . finish ( ) ; return BytesInput . from ( out ) ; } catch ( IOException e ) { throw new ParquetEncodingException ( e ) ; } }


Input: Factory method for record filter which applies the supplied predicate to the specified column . Note that if searching for a repeated sub - attribute it will only ever match against the first instance of it in the object . [CODESPLIT] public static final UnboundRecordFilter column ( final String columnPath , final ColumnPredicates . Predicate predicate ) { checkNotNull ( columnPath , "columnPath" ) ; checkNotNull ( predicate , "predicate" ) ; return new UnboundRecordFilter ( ) { final String [ ] filterPath = columnPath . split ( "\\." ) ; @ Override public RecordFilter bind ( Iterable < ColumnReader > readers ) { for ( ColumnReader reader : readers ) { if ( Arrays . equals ( reader . getDescriptor ( ) . getPath ( ) , filterPath ) ) { return new ColumnRecordFilter ( reader , predicate ) ; } } throw new IllegalArgumentException ( "Column " + columnPath + " does not exist." ) ; } } ; }


Input: / * Returns the filtered offset index containing only the pages which are overlapping with rowRanges . [CODESPLIT] static OffsetIndex filterOffsetIndex ( OffsetIndex offsetIndex , RowRanges rowRanges , long totalRowCount ) { IntList indexMap = new IntArrayList ( ) ; for ( int i = 0 , n = offsetIndex . getPageCount ( ) ; i < n ; ++ i ) { long from = offsetIndex . getFirstRowIndex ( i ) ; if ( rowRanges . isOverlapping ( from , offsetIndex . getLastRowIndex ( i , totalRowCount ) ) ) { indexMap . add ( i ) ; } } return new FilteredOffsetIndex ( offsetIndex , indexMap . toIntArray ( ) ) ; }


Input: struct is assumed to contain valid structOrUnionType metadata when used with this method . This method may throw if structOrUnionType is unknown . [CODESPLIT] public MessageType convert ( StructType struct ) { MessageType messageType = ThriftSchemaConvertVisitor . convert ( struct , fieldProjectionFilter , true ) ; fieldProjectionFilter . assertNoUnmatchedPatterns ( ) ; return messageType ; }


Input: struct is not required to have known structOrUnionType which is useful for converting a StructType from an ( older ) file schema to a MessageType [CODESPLIT] public static MessageType convertWithoutProjection ( StructType struct ) { return ThriftSchemaConvertVisitor . convert ( struct , FieldProjectionFilter . ALL_COLUMNS , false ) ; }


Input: Returns whether the given type is the element type of a list or is a synthetic group with one field that is the element type . This is determined by checking whether the type can be a synthetic group and by checking whether a potential synthetic group matches the expected ThriftField . <p > This method never guesses because the expected ThriftField is known . [CODESPLIT] static boolean isListElementType ( Type repeatedType , ThriftField thriftElement ) { if ( repeatedType . isPrimitive ( ) || ( repeatedType . asGroupType ( ) . getFieldCount ( ) != 1 ) || ( repeatedType . asGroupType ( ) . getType ( 0 ) . isRepetition ( REPEATED ) ) ) { // The repeated type must be the element type because it is an invalid // synthetic wrapper. Must be a group with one optional or required field return true ; } else if ( thriftElement != null && thriftElement . getType ( ) instanceof StructType ) { Set < String > fieldNames = new HashSet < String > ( ) ; for ( ThriftField field : ( ( StructType ) thriftElement . getType ( ) ) . getChildren ( ) ) { fieldNames . add ( field . getName ( ) ) ; } // If the repeated type is a subset of the structure of the ThriftField, // then it must be the element type. return fieldNames . contains ( repeatedType . asGroupType ( ) . getFieldName ( 0 ) ) ; } return false ; }


Input: to preserve the difference between empty list and null when optional [CODESPLIT] private static GroupType listWrapper ( Repetition repetition , String alias , LogicalTypeAnnotation logicalTypeAnnotation , Type nested ) { if ( ! nested . isRepetition ( Repetition . REPEATED ) ) { throw new IllegalArgumentException ( "Nested type should be repeated: " + nested ) ; } return new GroupType ( repetition , alias , logicalTypeAnnotation , nested ) ; }


Input: Creates a 3 - level list structure annotated with LIST with elements of the given elementType . The repeated level is inserted automatically and the elementType s repetition should be the correct repetition of the elements required for non - null and optional for nullable . [CODESPLIT] public static GroupType listOfElements ( Repetition listRepetition , String name , Type elementType ) { Preconditions . checkArgument ( elementType . getName ( ) . equals ( ELEMENT_NAME ) , "List element type must be named 'element'" ) ; return listWrapper ( listRepetition , name , LogicalTypeAnnotation . listType ( ) , new GroupType ( Repetition . REPEATED , "list" , elementType ) ) ; }


Input: { [CODESPLIT] @ Override public void checkSchema ( ResourceSchema s ) throws IOException { getProperties ( ) . setProperty ( SCHEMA , s . toString ( ) ) ; }


Input: { [CODESPLIT] @ Override public OutputFormat < Void , Tuple > getOutputFormat ( ) throws IOException { Schema pigSchema = getSchema ( ) ; return new ParquetOutputFormat < Tuple > ( new TupleWriteSupport ( pigSchema ) ) ; }


Input: { [CODESPLIT] @ Override public void putNext ( Tuple tuple ) throws IOException { try { this . recordWriter . write ( null , tuple ) ; } catch ( InterruptedException e ) { Thread . interrupted ( ) ; throw new ParquetEncodingException ( "Interrupted while writing" , e ) ; } }


Input: If there is a conflicting value when reading from multiple files an exception will be thrown [CODESPLIT] @ Deprecated public Map < String , String > getMergedKeyValueMetaData ( ) { if ( mergedKeyValueMetadata == null ) { Map < String , String > mergedKeyValues = new HashMap < String , String > ( ) ; for ( Entry < String , Set < String > > entry : keyValueMetadata . entrySet ( ) ) { if ( entry . getValue ( ) . size ( ) > 1 ) { throw new RuntimeException ( "could not merge metadata: key " + entry . getKey ( ) + " has conflicting values: " + entry . getValue ( ) ) ; } mergedKeyValues . put ( entry . getKey ( ) , entry . getValue ( ) . iterator ( ) . next ( ) ) ; } mergedKeyValueMetadata = mergedKeyValues ; } return mergedKeyValueMetadata ; }


Input: gets a ParquetInputSplit corresponding to a split given by Hive [CODESPLIT] protected ParquetInputSplit getSplit ( final InputSplit oldSplit , final JobConf conf ) throws IOException { if ( oldSplit instanceof FileSplit ) { FileSplit fileSplit = ( FileSplit ) oldSplit ; final long splitStart = fileSplit . getStart ( ) ; final long splitLength = fileSplit . getLength ( ) ; final Path finalPath = fileSplit . getPath ( ) ; final JobConf cloneJob = hiveBinding . pushProjectionsAndFilters ( conf , finalPath . getParent ( ) ) ; final ParquetMetadata parquetMetadata = ParquetFileReader . readFooter ( cloneJob , finalPath , SKIP_ROW_GROUPS ) ; final FileMetaData fileMetaData = parquetMetadata . getFileMetaData ( ) ; final ReadContext readContext = new DataWritableReadSupport ( ) . init ( cloneJob , fileMetaData . getKeyValueMetaData ( ) , fileMetaData . getSchema ( ) ) ; schemaSize = MessageTypeParser . parseMessageType ( readContext . getReadSupportMetadata ( ) . get ( DataWritableReadSupport . HIVE_SCHEMA_KEY ) ) . getFieldCount ( ) ; return new ParquetInputSplit ( finalPath , splitStart , splitStart + splitLength , splitLength , fileSplit . getLocations ( ) , null ) ; } else { throw new IllegalArgumentException ( "Unknown split type: " + oldSplit ) ; } }


Input: { [CODESPLIT] @ Override public RecordWriter < Void , T > getRecordWriter ( TaskAttemptContext taskAttemptContext ) throws IOException , InterruptedException { final Configuration conf = getConfiguration ( taskAttemptContext ) ; CompressionCodecName codec = getCodec ( taskAttemptContext ) ; String extension = codec . getExtension ( ) + ".parquet" ; Path file = getDefaultWorkFile ( taskAttemptContext , extension ) ; return getRecordWriter ( conf , file , codec ) ; }


Input: See the general contract of the <code > skipBytes< / code > method of <code > DataInput< / code > . <p > Bytes for this operation are read from the contained input stream . [CODESPLIT] public final int skipBytes ( int n ) throws IOException { int total = 0 ; int cur = 0 ; while ( ( total < n ) && ( ( cur = ( int ) in . skip ( n - total ) ) > 0 ) ) { total += cur ; } return total ; }


Input: Bytes for this operation are read from the contained input stream . [CODESPLIT] public final int readUnsignedShort ( ) throws IOException { int ch2 = in . read ( ) ; int ch1 = in . read ( ) ; if ( ( ch1 | ch2 ) < 0 ) throw new EOFException ( ) ; return ( ch1 << 8 ) + ( ch2 << 0 ) ; }


Input: Bytes for this operation are read from the contained input stream . [CODESPLIT] public final long readLong ( ) throws IOException { // TODO: see perf question above in readInt readFully ( readBuffer , 0 , 8 ) ; return ( ( ( long ) readBuffer [ 7 ] << 56 ) + ( ( long ) ( readBuffer [ 6 ] & 255 ) << 48 ) + ( ( long ) ( readBuffer [ 5 ] & 255 ) << 40 ) + ( ( long ) ( readBuffer [ 4 ] & 255 ) << 32 ) + ( ( long ) ( readBuffer [ 3 ] & 255 ) << 24 ) + ( ( readBuffer [ 2 ] & 255 ) << 16 ) + ( ( readBuffer [ 1 ] & 255 ) << 8 ) + ( ( readBuffer [ 0 ] & 255 ) << 0 ) ) ; }


Input: Builds a { @code ParquetInputSplit } from a mapreduce { @link FileSplit } . [CODESPLIT] static ParquetInputSplit from ( FileSplit split ) throws IOException { return new ParquetInputSplit ( split . getPath ( ) , split . getStart ( ) , split . getStart ( ) + split . getLength ( ) , split . getLength ( ) , split . getLocations ( ) , null ) ; }


Input: Builds a { @code ParquetInputSplit } from a mapred { @link org . apache . hadoop . mapred . FileSplit } . [CODESPLIT] static ParquetInputSplit from ( org . apache . hadoop . mapred . FileSplit split ) throws IOException { return new ParquetInputSplit ( split . getPath ( ) , split . getStart ( ) , split . getStart ( ) + split . getLength ( ) , split . getLength ( ) , split . getLocations ( ) , null ) ; }


Input: { [CODESPLIT] @ Override public void readFields ( DataInput hin ) throws IOException { byte [ ] bytes = readArray ( hin ) ; DataInputStream in = new DataInputStream ( new GZIPInputStream ( new ByteArrayInputStream ( bytes ) ) ) ; super . readFields ( in ) ; this . end = in . readLong ( ) ; if ( in . readBoolean ( ) ) { this . rowGroupOffsets = new long [ in . readInt ( ) ] ; for ( int i = 0 ; i < rowGroupOffsets . length ; i ++ ) { rowGroupOffsets [ i ] = in . readLong ( ) ; } } in . close ( ) ; }


Input: { [CODESPLIT] @ Override public void write ( DataOutput hout ) throws IOException { ByteArrayOutputStream baos = new ByteArrayOutputStream ( ) ; DataOutputStream out = new DataOutputStream ( new GZIPOutputStream ( baos ) ) ; super . write ( out ) ; out . writeLong ( end ) ; out . writeBoolean ( rowGroupOffsets != null ) ; if ( rowGroupOffsets != null ) { out . writeInt ( rowGroupOffsets . length ) ; for ( long o : rowGroupOffsets ) { out . writeLong ( o ) ; } } out . close ( ) ; writeArray ( hout , baos . toByteArray ( ) ) ; }


Input: Returns a builder to construct a required { @link PrimitiveType } . [CODESPLIT] public static PrimitiveBuilder < PrimitiveType > required ( PrimitiveTypeName type ) { return new PrimitiveBuilder < PrimitiveType > ( PrimitiveType . class , type ) . repetition ( Type . Repetition . REQUIRED ) ; }


Input: Returns a builder to construct an optional { @link PrimitiveType } . [CODESPLIT] public static PrimitiveBuilder < PrimitiveType > optional ( PrimitiveTypeName type ) { return new PrimitiveBuilder < PrimitiveType > ( PrimitiveType . class , type ) . repetition ( Type . Repetition . OPTIONAL ) ; }


Input: Returns a builder to construct a repeated { @link PrimitiveType } . [CODESPLIT] public static PrimitiveBuilder < PrimitiveType > repeated ( PrimitiveTypeName type ) { return new PrimitiveBuilder < PrimitiveType > ( PrimitiveType . class , type ) . repetition ( Type . Repetition . REPEATED ) ; }


Input: Returns a builder to construct a required { @link GroupType } . [CODESPLIT] public static GroupBuilder < GroupType > requiredGroup ( ) { return new GroupBuilder < GroupType > ( GroupType . class ) . repetition ( Type . Repetition . REQUIRED ) ; }


Input: Returns a builder to construct an optional { @link GroupType } . [CODESPLIT] public static GroupBuilder < GroupType > optionalGroup ( ) { return new GroupBuilder < GroupType > ( GroupType . class ) . repetition ( Type . Repetition . OPTIONAL ) ; }


Input: Returns a builder to construct a repeated { @link GroupType } . [CODESPLIT] public static GroupBuilder < GroupType > repeatedGroup ( ) { return new GroupBuilder < GroupType > ( GroupType . class ) . repetition ( Type . Repetition . REPEATED ) ; }


Input: ( not an actual Iterable ) [CODESPLIT] public IntIterator iterator ( ) { if ( currentSlab == null ) { allocateSlab ( ) ; } int [ ] [ ] itSlabs = slabs . toArray ( new int [ slabs . size ( ) + 1 ] [  ] ) ; itSlabs [ slabs . size ( ) ] = currentSlab ; return new IntIterator ( itSlabs , size ( ) ) ; }


Input: Decides if the statistics from a file created by createdBy ( the created_by field from parquet format ) should be ignored because they are potentially corrupt . [CODESPLIT] public static boolean shouldIgnoreStatistics ( String createdBy , PrimitiveTypeName columnType ) { if ( columnType != PrimitiveTypeName . BINARY && columnType != PrimitiveTypeName . FIXED_LEN_BYTE_ARRAY ) { // the bug only applies to binary columns return false ; } if ( Strings . isNullOrEmpty ( createdBy ) ) { // created_by is not populated, which could have been caused by // parquet-mr during the same time as PARQUET-251, see PARQUET-297 warnOnce ( "Ignoring statistics because created_by is null or empty! See PARQUET-251 and PARQUET-297" ) ; return true ; } try { ParsedVersion version = VersionParser . parse ( createdBy ) ; if ( ! "parquet-mr" . equals ( version . application ) ) { // assume other applications don't have this bug return false ; } if ( Strings . isNullOrEmpty ( version . version ) ) { warnOnce ( "Ignoring statistics because created_by did not contain a semver (see PARQUET-251): " + createdBy ) ; return true ; } SemanticVersion semver = SemanticVersion . parse ( version . version ) ; if ( semver . compareTo ( PARQUET_251_FIXED_VERSION ) < 0 && ! ( semver . compareTo ( CDH_5_PARQUET_251_FIXED_START ) >= 0 && semver . compareTo ( CDH_5_PARQUET_251_FIXED_END ) < 0 ) ) { warnOnce ( "Ignoring statistics because this file was created prior to " + PARQUET_251_FIXED_VERSION + ", see PARQUET-251" ) ; return true ; } // this file was created after the fix return false ; } catch ( RuntimeException e ) { // couldn't parse the created_by field, log what went wrong, don't trust the stats, // but don't make this fatal. warnParseErrorOnce ( createdBy , e ) ; return true ; } catch ( SemanticVersionParseException e ) { // couldn't parse the created_by field, log what went wrong, don't trust the stats, // but don't make this fatal. warnParseErrorOnce ( createdBy , e ) ; return true ; } catch ( VersionParseException e ) { // couldn't parse the created_by field, log what went wrong, don't trust the stats, // but don't make this fatal. warnParseErrorOnce ( createdBy , e ) ; return true ; } }


Input: TODO : have those wrappers for a converter [CODESPLIT] private RecordConsumer validator ( RecordConsumer recordConsumer , boolean validating , MessageType schema ) { return validating ? new ValidatingRecordConsumer ( recordConsumer , schema ) : recordConsumer ; }


Input: this was taken from Avro s ReflectData [CODESPLIT] private static Map < String , Class < ? > > getFieldsByName ( Class < ? > recordClass , boolean excludeJava ) { Map < String , Class < ? > > fields = new LinkedHashMap < String , Class < ? > > ( ) ; if ( recordClass != null ) { Class < ? > current = recordClass ; do { if ( excludeJava && current . getPackage ( ) != null && current . getPackage ( ) . getName ( ) . startsWith ( "java." ) ) { break ; // skip java built-in classes } for ( Field field : current . getDeclaredFields ( ) ) { if ( field . isAnnotationPresent ( AvroIgnore . class ) || isTransientOrStatic ( field ) ) { continue ; } AvroName altName = field . getAnnotation ( AvroName . class ) ; Class < ? > existing = fields . put ( altName != null ? altName . value ( ) : field . getName ( ) , field . getType ( ) ) ; if ( existing != null ) { throw new AvroTypeException ( current + " contains two fields named: " + field . getName ( ) ) ; } } current = current . getSuperclass ( ) ; } while ( current != null ) ; } return fields ; }


Input: Returns whether the given type is the element type of a list or is a synthetic group with one field that is the element type . This is determined by checking whether the type can be a synthetic group and by checking whether a potential synthetic group matches the expected schema . <p > Unlike { @link AvroSchemaConverter#isElementType ( Type String ) } this method never guesses because the expected schema is known . [CODESPLIT] static boolean isElementType ( Type repeatedType , Schema elementSchema ) { if ( repeatedType . isPrimitive ( ) || repeatedType . asGroupType ( ) . getFieldCount ( ) > 1 || repeatedType . asGroupType ( ) . getType ( 0 ) . isRepetition ( REPEATED ) ) { // The repeated type must be the element type because it is an invalid // synthetic wrapper. Must be a group with one optional or required field return true ; } else if ( elementSchema != null && elementSchema . getType ( ) == Schema . Type . RECORD ) { Schema schemaFromRepeated = CONVERTER . convert ( repeatedType . asGroupType ( ) ) ; if ( checkReaderWriterCompatibility ( elementSchema , schemaFromRepeated ) . getType ( ) == COMPATIBLE ) { return true ; } } return false ; }


Input: { [CODESPLIT] @ Override public void writeMessageBegin ( TMessage message ) throws TException { LOG . debug ( "writeMessageBegin({})" , message ) ; currentProtocol . writeMessageBegin ( message ) ; }


Input: { [CODESPLIT] @ Override public void writeStructBegin ( TStruct struct ) throws TException { if ( LOG . isDebugEnabled ( ) ) LOG . debug ( "writeStructBegin(" + toString ( struct ) + ")" ) ; currentProtocol . writeStructBegin ( struct ) ; }


Input: { [CODESPLIT] @ Override public void writeFieldBegin ( TField field ) throws TException { LOG . debug ( "writeFieldBegin({})" , field ) ; currentProtocol . writeFieldBegin ( field ) ; }


Input: { [CODESPLIT] @ Override public void writeMapBegin ( TMap map ) throws TException { if ( LOG . isDebugEnabled ( ) ) LOG . debug ( "writeMapBegin(" + toString ( map ) + ")" ) ; currentProtocol . writeMapBegin ( map ) ; }


Input: { [CODESPLIT] @ Override public void writeListBegin ( TList list ) throws TException { if ( LOG . isDebugEnabled ( ) ) LOG . debug ( "writeListBegin(" + toString ( list ) + ")" ) ; currentProtocol . writeListBegin ( list ) ; }


Input: { [CODESPLIT] @ Override public void writeSetBegin ( TSet set ) throws TException { LOG . debug ( "writeSetBegin({})" , set ) ; currentProtocol . writeSetBegin ( set ) ; }


Input: { [CODESPLIT] @ Override public void writeBool ( boolean b ) throws TException { LOG . debug ( "writeBool({})" , b ) ; currentProtocol . writeBool ( b ) ; }


Input: { [CODESPLIT] @ Override public void writeByte ( byte b ) throws TException { LOG . debug ( "writeByte({})" , b ) ; currentProtocol . writeByte ( b ) ; }


Input: { [CODESPLIT] @ Override public void writeI16 ( short i16 ) throws TException { LOG . debug ( "writeI16({})" , i16 ) ; currentProtocol . writeI16 ( i16 ) ; }


Input: { [CODESPLIT] @ Override public void writeI32 ( int i32 ) throws TException { LOG . debug ( "writeI32({})" , i32 ) ; currentProtocol . writeI32 ( i32 ) ; }


Input: { [CODESPLIT] @ Override public void writeI64 ( long i64 ) throws TException { LOG . debug ( "writeI64({})" , i64 ) ; currentProtocol . writeI64 ( i64 ) ; }


Input: { [CODESPLIT] @ Override public void writeDouble ( double dub ) throws TException { LOG . debug ( "writeDouble({})" , dub ) ; currentProtocol . writeDouble ( dub ) ; }


Input: { [CODESPLIT] @ Override public void writeString ( String str ) throws TException { LOG . debug ( "writeString({})" , str ) ; currentProtocol . writeString ( str ) ; }


Input: { [CODESPLIT] @ Override public void writeBinary ( ByteBuffer buf ) throws TException { LOG . debug ( "writeBinary({})" , buf ) ; currentProtocol . writeBinary ( buf ) ; }


Input: Determine the file column names based on the position within the requested columns and use that as the requested schema . [CODESPLIT] private MessageType resolveSchemaAccess ( MessageType requestedSchema , MessageType fileSchema , Configuration configuration ) { if ( configuration . getBoolean ( PARQUET_COLUMN_INDEX_ACCESS , false ) ) { final List < String > listColumns = getColumns ( configuration . get ( IOConstants . COLUMNS ) ) ; List < Type > requestedTypes = new ArrayList < Type > ( ) ; for ( Type t : requestedSchema . getFields ( ) ) { int index = listColumns . indexOf ( t . getName ( ) ) ; requestedTypes . add ( fileSchema . getType ( index ) ) ; } requestedSchema = new MessageType ( requestedSchema . getName ( ) , requestedTypes ) ; } return requestedSchema ; }


Input: { [CODESPLIT] @ Override public void startField ( String field , int index ) { logOpen ( field ) ; delegate . startField ( field , index ) ; }


Input: { [CODESPLIT] @ Override public void addBinary ( Binary value ) { if ( LOG . isDebugEnabled ( ) ) log ( Arrays . toString ( value . getBytesUnsafe ( ) ) ) ; delegate . addBinary ( value ) ; }


Input: { [CODESPLIT] @ Override public void endField ( String field , int index ) { logClose ( field ) ; delegate . endField ( field , index ) ; }


Input: Get all input files . [CODESPLIT] private List < Path > getInputFiles ( List < String > input ) throws IOException { List < Path > inputFiles = null ; if ( input . size ( ) == 1 ) { Path p = new Path ( input . get ( 0 ) ) ; FileSystem fs = p . getFileSystem ( conf ) ; FileStatus status = fs . getFileStatus ( p ) ; if ( status . isDir ( ) ) { inputFiles = getInputFilesFromDirectory ( status ) ; } } else { inputFiles = parseInputFiles ( input ) ; } checkParquetFiles ( inputFiles ) ; return inputFiles ; }


Input: Check input files basically . ParquetFileReader will throw exception when reading an illegal parquet file . [CODESPLIT] private void checkParquetFiles ( List < Path > inputFiles ) throws IOException { if ( inputFiles == null || inputFiles . size ( ) <= 1 ) { throw new IllegalArgumentException ( "Not enough files to merge" ) ; } for ( Path inputFile : inputFiles ) { FileSystem fs = inputFile . getFileSystem ( conf ) ; FileStatus status = fs . getFileStatus ( inputFile ) ; if ( status . isDir ( ) ) { throw new IllegalArgumentException ( "Illegal parquet file: " + inputFile . toUri ( ) ) ; } } }


Input: Get all parquet files under partition directory . [CODESPLIT] private List < Path > getInputFilesFromDirectory ( FileStatus partitionDir ) throws IOException { FileSystem fs = partitionDir . getPath ( ) . getFileSystem ( conf ) ; FileStatus [ ] inputFiles = fs . listStatus ( partitionDir . getPath ( ) , HiddenFileFilter . INSTANCE ) ; List < Path > input = new ArrayList < Path > ( ) ; for ( FileStatus f : inputFiles ) { input . add ( f . getPath ( ) ) ; } return input ; }


Input: { [CODESPLIT] @ Override public void initFromPage ( int valueCount , ByteBufferInputStream stream ) throws IOException { int effectiveBitLength = valueCount * bitsPerValue ; int length = BytesUtils . paddedByteCountFromBits ( effectiveBitLength ) ; LOG . debug ( "reading {} bytes for {} values of size {} bits." , length , valueCount , bitsPerValue ) ; this . in = stream . sliceStream ( length ) ; this . bitPackingReader = createBitPackingReader ( bitsPerValue , this . in , valueCount ) ; updateNextOffset ( length ) ; }


Input: Returns builder for creating a paged query . [CODESPLIT] public static final UnboundRecordFilter page ( final long startPos , final long pageSize ) { return new UnboundRecordFilter ( ) { @ Override public RecordFilter bind ( Iterable < ColumnReader > readers ) { return new PagedRecordFilter ( startPos , pageSize ) ; } } ; }


Input: Visible for testing [CODESPLIT] static void readFully ( InputStream f , byte [ ] bytes , int start , int len ) throws IOException { int offset = start ; int remaining = len ; while ( remaining > 0 ) { int bytesRead = f . read ( bytes , offset , remaining ) ; if ( bytesRead < 0 ) { throw new EOFException ( "Reached the end of stream with " + remaining + " bytes left to read" ) ; } remaining -= bytesRead ; offset += bytesRead ; } }


Input: Visible for testing [CODESPLIT] static int readHeapBuffer ( InputStream f , ByteBuffer buf ) throws IOException { int bytesRead = f . read ( buf . array ( ) , buf . arrayOffset ( ) + buf . position ( ) , buf . remaining ( ) ) ; if ( bytesRead < 0 ) { // if this resulted in EOF, don't update position return bytesRead ; } else { buf . position ( buf . position ( ) + bytesRead ) ; return bytesRead ; } }


Input: Visible for testing [CODESPLIT] static void readFullyHeapBuffer ( InputStream f , ByteBuffer buf ) throws IOException { readFully ( f , buf . array ( ) , buf . arrayOffset ( ) + buf . position ( ) , buf . remaining ( ) ) ; buf . position ( buf . limit ( ) ) ; }


Input: Visible for testing [CODESPLIT] static int readDirectBuffer ( InputStream f , ByteBuffer buf , byte [ ] temp ) throws IOException { // copy all the bytes that return immediately, stopping at the first // read that doesn't return a full buffer. int nextReadLength = Math . min ( buf . remaining ( ) , temp . length ) ; int totalBytesRead = 0 ; int bytesRead ; while ( ( bytesRead = f . read ( temp , 0 , nextReadLength ) ) == temp . length ) { buf . put ( temp ) ; totalBytesRead += bytesRead ; nextReadLength = Math . min ( buf . remaining ( ) , temp . length ) ; } if ( bytesRead < 0 ) { // return -1 if nothing was read return totalBytesRead == 0 ? - 1 : totalBytesRead ; } else { // copy the last partial buffer buf . put ( temp , 0 , bytesRead ) ; totalBytesRead += bytesRead ; return totalBytesRead ; } }


Input: Visible for testing [CODESPLIT] static void readFullyDirectBuffer ( InputStream f , ByteBuffer buf , byte [ ] temp ) throws IOException { int nextReadLength = Math . min ( buf . remaining ( ) , temp . length ) ; int bytesRead = 0 ; while ( nextReadLength > 0 && ( bytesRead = f . read ( temp , 0 , nextReadLength ) ) >= 0 ) { buf . put ( temp , 0 , bytesRead ) ; nextReadLength = Math . min ( buf . remaining ( ) , temp . length ) ; } if ( bytesRead < 0 && buf . remaining ( ) > 0 ) { throw new EOFException ( "Reached the end of stream with " + buf . remaining ( ) + " bytes left to read" ) ; } }


Input: To consume a list of elements [CODESPLIT] public static < T extends TBase < T , ? extends TFieldIdEnum > > ListConsumer listOf ( Class < T > c , final Consumer < List < T > > consumer ) { class ListConsumer implements Consumer < T > { List < T > list ; @ Override public void consume ( T t ) { list . add ( t ) ; } } final ListConsumer co = new ListConsumer ( ) ; return new DelegatingListElementsConsumer ( struct ( c , co ) ) { @ Override public void consumeList ( TProtocol protocol , EventBasedThriftReader reader , TList tList ) throws TException { co . list = new ArrayList < T > ( ) ; super . consumeList ( protocol , reader , tList ) ; consumer . consume ( co . list ) ; } } ; }


Input: Initialize the mrwork variable in order to get all the partition and start to update the jobconf [CODESPLIT] private void init ( final JobConf job ) { final String plan = HiveConf . getVar ( job , HiveConf . ConfVars . PLAN ) ; if ( mrwork == null && plan != null && plan . length ( ) > 0 ) { mrwork = Utilities . getMapRedWork ( job ) ; pathToPartitionInfo . clear ( ) ; for ( final Map . Entry < String , PartitionDesc > entry : mrwork . getPathToPartitionInfo ( ) . entrySet ( ) ) { pathToPartitionInfo . put ( new Path ( entry . getKey ( ) ) . toUri ( ) . getPath ( ) . toString ( ) , entry . getValue ( ) ) ; } } }


Input: { [CODESPLIT] @ Override public JobConf pushProjectionsAndFilters ( JobConf jobConf , Path path ) throws IOException { init ( jobConf ) ; final JobConf cloneJobConf = new JobConf ( jobConf ) ; final PartitionDesc part = pathToPartitionInfo . get ( path . toString ( ) ) ; if ( ( part != null ) && ( part . getTableDesc ( ) != null ) ) { Utilities . copyTableJobPropertiesToConf ( part . getTableDesc ( ) , cloneJobConf ) ; } pushProjectionsAndFilters ( cloneJobConf , path . toString ( ) , path . toUri ( ) . toString ( ) ) ; return cloneJobConf ; }


Input: / * Iterates over list of fields . * [CODESPLIT] private < T > GroupBuilder < T > convertFields ( GroupBuilder < T > groupBuilder , List < FieldDescriptor > fieldDescriptors ) { for ( FieldDescriptor fieldDescriptor : fieldDescriptors ) { groupBuilder = addField ( fieldDescriptor , groupBuilder ) . id ( fieldDescriptor . getNumber ( ) ) . named ( fieldDescriptor . getName ( ) ) ; } return groupBuilder ; }


Input: filters a Parquet schema based on a pig schema for projection [CODESPLIT] public MessageType filter ( MessageType schemaToFilter , Schema requestedPigSchema ) { return filter ( schemaToFilter , requestedPigSchema , null ) ; }


Input: filters a Parquet schema based on a pig schema for projection [CODESPLIT] public MessageType filter ( MessageType schemaToFilter , Schema requestedPigSchema , RequiredFieldList requiredFieldList ) { try { if ( LOG . isDebugEnabled ( ) ) LOG . debug ( "filtering schema:\n" + schemaToFilter + "\nwith requested pig schema:\n " + requestedPigSchema ) ; List < Type > result = columnAccess . filterTupleSchema ( schemaToFilter , requestedPigSchema , requiredFieldList ) ; if ( LOG . isDebugEnabled ( ) ) LOG . debug ( "schema:\n" + schemaToFilter + "\nfiltered to:\n" + result ) ; return new MessageType ( schemaToFilter . getName ( ) , result ) ; } catch ( RuntimeException e ) { throw new RuntimeException ( "can't filter " + schemaToFilter + " with " + requestedPigSchema , e ) ; } }


Input: Create a { @link Schema } for the given type . If the type is null the schema will be a nullable String . If isNullable is true the returned schema will be nullable . [CODESPLIT] private static Schema schema ( Schema . Type type , boolean makeNullable ) { Schema schema = Schema . create ( type == null ? Schema . Type . STRING : type ) ; if ( makeNullable || type == null ) { schema = Schema . createUnion ( Lists . newArrayList ( Schema . create ( Schema . Type . NULL ) , schema ) ) ; } return schema ; }


Input: A { @link ThriftRecordConverter } builds an object by working with { @link TProtocol } . The default implementation creates standard Apache Thrift { @link TBase } objects ; to support alternatives such as <a href = http : // github . com / twitter / scrooge > Twiter s Scrooge< / a > a custom converter can be specified ( for example ScroogeRecordConverter from parquet - scrooge ) . [CODESPLIT] @ Deprecated public static void setRecordConverterClass ( JobConf conf , Class < ? > klass ) { setRecordConverterClass ( ( Configuration ) conf , klass ) ; }


Input: A { @link ThriftRecordConverter } builds an object by working with { @link TProtocol } . The default implementation creates standard Apache Thrift { @link TBase } objects ; to support alternatives such as <a href = http : // github . com / twitter / scrooge > Twiter s Scrooge< / a > a custom converter can be specified ( for example ScroogeRecordConverter from parquet - scrooge ) . [CODESPLIT] public static void setRecordConverterClass ( Configuration conf , Class < ? > klass ) { conf . set ( RECORD_CONVERTER_CLASS_KEY , klass . getName ( ) ) ; }


Input: the input tuple contains a bag of string representations of TupleSummaryData [CODESPLIT] private static TupleSummaryData merge ( Tuple t ) throws IOException { TupleSummaryData summaryData = new TupleSummaryData ( ) ; DataBag bag = ( DataBag ) t . get ( 0 ) ; for ( Tuple tuple : bag ) { summaryData . merge ( getData ( tuple ) ) ; } return summaryData ; }


Input: The input tuple contains a bag of Tuples to sum up [CODESPLIT] private static TupleSummaryData sumUp ( Schema schema , Tuple t ) throws ExecException { TupleSummaryData summaryData = new TupleSummaryData ( ) ; DataBag bag = ( DataBag ) t . get ( 0 ) ; for ( Tuple tuple : bag ) { summaryData . addTuple ( schema , tuple ) ; } return summaryData ; }


Input: { [CODESPLIT] @ Override public void initialize ( InputSplit inputSplit , TaskAttemptContext context ) throws IOException , InterruptedException { if ( ContextUtil . hasCounterMethod ( context ) ) { BenchmarkCounter . initCounterFromContext ( context ) ; } else { LOG . error ( String . format ( "Can not initialize counter because the class '%s' does not have a '.getCounterMethod'" , context . getClass ( ) . getCanonicalName ( ) ) ) ; } initializeInternalReader ( toParquetSplit ( inputSplit ) , ContextUtil . getConfiguration ( context ) ) ; }


Input: } [CODESPLIT] @ Benchmark public void write1MRowsDefaultBlockAndPageSizeSNAPPY ( ) throws IOException { dataGenerator . generateData ( file_1M_SNAPPY , configuration , PARQUET_2_0 , BLOCK_SIZE_DEFAULT , PAGE_SIZE_DEFAULT , FIXED_LEN_BYTEARRAY_SIZE , SNAPPY , ONE_MILLION ) ; }


Input: reads a Struct from the underlying protocol and passes the field events to the FieldConsumer [CODESPLIT] public void readStruct ( FieldConsumer c ) throws TException { protocol . readStructBegin ( ) ; readStructContent ( c ) ; protocol . readStructEnd ( ) ; }


Input: reads the content of a struct ( fields ) from the underlying protocol and passes the events to c [CODESPLIT] public void readStructContent ( FieldConsumer c ) throws TException { TField field ; while ( true ) { field = protocol . readFieldBegin ( ) ; if ( field . type == TType . STOP ) { break ; } c . consumeField ( protocol , this , field . id , field . type ) ; } }


Input: reads the set content ( elements ) from the underlying protocol and passes the events to the set event consumer [CODESPLIT] public void readSetContent ( SetConsumer eventConsumer , TSet tSet ) throws TException { for ( int i = 0 ; i < tSet . size ; i ++ ) { eventConsumer . consumeElement ( protocol , this , tSet . elemType ) ; } }


Input: reads the map content ( key values ) from the underlying protocol and passes the events to the map event consumer [CODESPLIT] public void readMapContent ( MapConsumer eventConsumer , TMap tMap ) throws TException { for ( int i = 0 ; i < tMap . size ; i ++ ) { eventConsumer . consumeEntry ( protocol , this , tMap . keyType , tMap . valueType ) ; } }


Input: reads a key - value pair [CODESPLIT] public void readMapEntry ( byte keyType , TypedConsumer keyConsumer , byte valueType , TypedConsumer valueConsumer ) throws TException { keyConsumer . read ( protocol , this , keyType ) ; valueConsumer . read ( protocol , this , valueType ) ; }


Input: reads the list content ( elements ) from the underlying protocol and passes the events to the list event consumer [CODESPLIT] public void readListContent ( ListConsumer eventConsumer , TList tList ) throws TException { for ( int i = 0 ; i < tList . size ; i ++ ) { eventConsumer . consumeElement ( protocol , this , tList . elemType ) ; } }


Input: writes an int using the requested number of bits . accepts only values less than 2^bitWidth [CODESPLIT] public void writeInt ( int value ) throws IOException { input [ inputSize ] = value ; ++ inputSize ; if ( inputSize == VALUES_WRITTEN_AT_A_TIME ) { pack ( ) ; if ( packedPosition == slabSize ) { slabs . add ( BytesInput . from ( packed ) ) ; totalFullSlabSize += slabSize ; if ( slabSize < bitWidth * MAX_SLAB_SIZE_MULT ) { slabSize *= 2 ; } initPackedSlab ( ) ; } } }


Input: reads an int in little endian at the given position [CODESPLIT] public static int readIntLittleEndian ( ByteBuffer in , int offset ) throws IOException { int ch4 = in . get ( offset ) & 0xff ; int ch3 = in . get ( offset + 1 ) & 0xff ; int ch2 = in . get ( offset + 2 ) & 0xff ; int ch1 = in . get ( offset + 3 ) & 0xff ; return ( ( ch1 << 24 ) + ( ch2 << 16 ) + ( ch3 << 8 ) + ( ch4 << 0 ) ) ; }


Input: Write a little endian int to out using the the number of bytes required by bit width @param out an output stream @param v an int value @param bitWidth bit width for padding @throws IOException if there is an exception while writing [CODESPLIT] public static void writeIntLittleEndianPaddedOnBitWidth ( OutputStream out , int v , int bitWidth ) throws IOException { int bytesWidth = paddedByteCountFromBits ( bitWidth ) ; switch ( bytesWidth ) { case 0 : break ; case 1 : writeIntLittleEndianOnOneByte ( out , v ) ; break ; case 2 : writeIntLittleEndianOnTwoBytes ( out , v ) ; break ; case 3 : writeIntLittleEndianOnThreeBytes ( out , v ) ; break ; case 4 : writeIntLittleEndian ( out , v ) ; break ; default : throw new IOException ( String . format ( "Encountered value (%d) that requires more than 4 bytes" , v ) ) ; } }


Input: uses a trick mentioned in https : // developers . google . com / protocol - buffers / docs / encoding to read zigZag encoded data [CODESPLIT] public static int readZigZagVarInt ( InputStream in ) throws IOException { int raw = readUnsignedVarInt ( in ) ; int temp = ( ( ( raw << 31 ) >> 31 ) ^ raw ) >> 1 ; return temp ^ ( raw & ( 1 << 31 ) ) ; }


Input: uses a trick mentioned in https : // developers . google . com / protocol - buffers / docs / encoding to read zigZag encoded data TODO : the implementation is compatible with readZigZagVarInt . Is there a need for different functions? [CODESPLIT] public static long readZigZagVarLong ( InputStream in ) throws IOException { long raw = readUnsignedVarLong ( in ) ; long temp = ( ( ( raw << 63 ) >> 63 ) ^ raw ) >> 1 ; return temp ^ ( raw & ( 1L << 63 ) ) ; }


Input: Fills specified buffer with uncompressed data . Returns actual number of bytes of uncompressed data . A return value of 0 indicates that { @link #needsInput () } should be called in order to determine if more input data is required . [CODESPLIT] @ Override public synchronized int decompress ( byte [ ] buffer , int off , int len ) throws IOException { SnappyUtil . validateBuffer ( buffer , off , len ) ; if ( inputBuffer . position ( ) == 0 && ! outputBuffer . hasRemaining ( ) ) { return 0 ; } if ( ! outputBuffer . hasRemaining ( ) ) { inputBuffer . rewind ( ) ; Preconditions . checkArgument ( inputBuffer . position ( ) == 0 , "Invalid position of 0." ) ; Preconditions . checkArgument ( outputBuffer . position ( ) == 0 , "Invalid position of 0." ) ; // There is compressed input, decompress it now. int decompressedSize = Snappy . uncompressedLength ( inputBuffer ) ; if ( decompressedSize > outputBuffer . capacity ( ) ) { ByteBuffer oldBuffer = outputBuffer ; outputBuffer = ByteBuffer . allocateDirect ( decompressedSize ) ; CleanUtil . clean ( oldBuffer ) ; } // Reset the previous outputBuffer (i.e. set position to 0) outputBuffer . clear ( ) ; int size = Snappy . uncompress ( inputBuffer , outputBuffer ) ; outputBuffer . limit ( size ) ; // We've decompressed the entire input, reset the input now inputBuffer . clear ( ) ; inputBuffer . limit ( 0 ) ; finished = true ; } // Return compressed output up to 'len' int numBytes = Math . min ( len , outputBuffer . remaining ( ) ) ; outputBuffer . get ( buffer , off , numBytes ) ; return numBytes ; }


Input: Sets input data for decompression . This should be called if and only if { @link #needsInput () } returns <code > true< / code > indicating that more input data is required . ( Both native and non - native versions of various Decompressors require that the data passed in via <code > b [] < / code > remain unmodified until the caller is explicitly notified -- via { @link #needsInput () } -- that the buffer may be safely modified . With this requirement an extra buffer - copy can be avoided . ) [CODESPLIT] @ Override public synchronized void setInput ( byte [ ] buffer , int off , int len ) { SnappyUtil . validateBuffer ( buffer , off , len ) ; if ( inputBuffer . capacity ( ) - inputBuffer . position ( ) < len ) { ByteBuffer newBuffer = ByteBuffer . allocateDirect ( inputBuffer . position ( ) + len ) ; inputBuffer . rewind ( ) ; newBuffer . put ( inputBuffer ) ; ByteBuffer oldBuffer = inputBuffer ; inputBuffer = newBuffer ; CleanUtil . clean ( oldBuffer ) ; } else { inputBuffer . limit ( inputBuffer . position ( ) + len ) ; } inputBuffer . put ( buffer , off , len ) ; }


Input: Set the Avro schema to use for writing . The schema is translated into a Parquet schema so that the records can be written in Parquet format . It is also stored in the Parquet metadata so that records can be reconstructed as Avro objects at read time without specifying a read schema . [CODESPLIT] public static void setSchema ( Job job , Schema schema ) { AvroWriteSupport . setSchema ( ContextUtil . getConfiguration ( job ) , schema ) ; }


Input: { [CODESPLIT] @ Override public List < String > getColumns ( final String columns ) { final List < String > result = ( List < String > ) StringUtils . getStringCollection ( columns ) ; result . removeAll ( virtualColumns ) ; return result ; }


Input: Wraps a { @link FSDataInputStream } in a { @link SeekableInputStream } implementation for Parquet readers . [CODESPLIT] public static SeekableInputStream wrap ( FSDataInputStream stream ) { Preconditions . checkNotNull ( stream , "Cannot wrap a null input stream" ) ; if ( byteBufferReadableClass != null && h2SeekableConstructor != null && byteBufferReadableClass . isInstance ( stream . getWrappedStream ( ) ) ) { try { return h2SeekableConstructor . newInstance ( stream ) ; } catch ( InstantiationException e ) { LOG . warn ( "Could not instantiate H2SeekableInputStream, falling back to byte array reads" , e ) ; return new H1SeekableInputStream ( stream ) ; } catch ( IllegalAccessException e ) { LOG . warn ( "Could not instantiate H2SeekableInputStream, falling back to byte array reads" , e ) ; return new H1SeekableInputStream ( stream ) ; } catch ( InvocationTargetException e ) { throw new ParquetDecodingException ( "Could not instantiate H2SeekableInputStream" , e . getTargetException ( ) ) ; } } else { return new H1SeekableInputStream ( stream ) ; } }


Input: theoretically they could just copy / paste the output into the Topics bulk edit field in the Kafka config [CODESPLIT] private < T > String convertToRawListString ( Collection < T > collection ) { return "[ " + collection . stream ( ) . map ( x -> ' ' + x . toString ( ) + ' ' ) . collect ( Collectors . joining ( ", " ) ) + " ]" ; }


Input: Should be called only by MapR Streams Producer . It creates a topic using KafkaProducer . [CODESPLIT] @ Override public void createTopicIfNotExists ( String topic , Map < String , Object > kafkaClientConfigs , String metadataBrokerList ) throws StageException { // check stream path and topic if ( topic . startsWith ( "/" ) && topic . contains ( ":" ) ) { String [ ] path = topic . split ( ":" ) ; if ( path . length != 2 ) { // Stream topic has invalid format. Record will be sent to error throw new StageException ( MapRStreamsErrors . MAPRSTREAMS_21 , topic ) ; } String streamPath = path [ 0 ] ; if ( ! streamCache . contains ( streamPath ) ) { // This pipeline sees this stream path for the 1st time Configuration conf = new Configuration ( ) ; kafkaClientConfigs . forEach ( ( k , v ) -> { conf . set ( k , v . toString ( ) ) ; } ) ; Admin streamAdmin = null ; try { streamAdmin = Streams . newAdmin ( conf ) ; // Check if the stream path exists already streamAdmin . countTopics ( streamPath ) ; streamCache . add ( streamPath ) ; } catch ( TableNotFoundException e ) { LOG . debug ( "Stream not found. Creating a new stream: " + streamPath ) ; try { streamAdmin . createStream ( streamPath , Streams . newStreamDescriptor ( ) ) ; streamCache . add ( streamPath ) ; } catch ( IOException ioex ) { throw new StageException ( MapRStreamsErrors . MAPRSTREAMS_22 , streamPath , e . getMessage ( ) , e ) ; } } catch ( IOException | IllegalArgumentException e ) { throw new StageException ( MapRStreamsErrors . MAPRSTREAMS_23 , e . getMessage ( ) , e ) ; } finally { if ( streamAdmin != null ) { streamAdmin . close ( ) ; } } } } // Stream topic can be created through KafkaProducer if Stream Path exists already KafkaProducer < String , String > kafkaProducer = createProducerTopicMetadataClient ( kafkaClientConfigs ) ; kafkaProducer . partitionsFor ( topic ) ; }


Input: Create a StringRedactor based on the JSON found in a file . The file format looks like this : { version : 1 rules : [ { description : This is the first rule trigger : triggerstring 1 search : regex 1 replace : replace 1 } { description : This is the second rule trigger : triggerstring 2 search : regex 2 replace : replace 2 } ] } [CODESPLIT] public static StringRedactor createFromJsonFile ( String fileName ) throws IOException { StringRedactor sr = new StringRedactor ( ) ; if ( fileName == null ) { sr . policy = RedactionPolicy . emptyRedactionPolicy ( ) ; return sr ; } File file = new File ( fileName ) ; // An empty file is explicitly allowed as "no rules" if ( file . exists ( ) && file . length ( ) == 0 ) { sr . policy = RedactionPolicy . emptyRedactionPolicy ( ) ; return sr ; } ObjectMapper mapper = new ObjectMapper ( ) ; RedactionPolicy policy = mapper . readValue ( file , RedactionPolicy . class ) ; policy . postProcess ( ) ; sr . policy = policy ; return sr ; }


Input: Create a StringRedactor based on the JSON found in the given String . The format is identical to that described in createFromJsonFile () . [CODESPLIT] public static StringRedactor createFromJsonString ( String json ) throws IOException { StringRedactor sr = new StringRedactor ( ) ; if ( ( json == null ) || json . isEmpty ( ) ) { sr . policy = RedactionPolicy . emptyRedactionPolicy ( ) ; return sr ; } ObjectMapper mapper = new ObjectMapper ( ) ; RedactionPolicy policy = mapper . readValue ( json , RedactionPolicy . class ) ; policy . postProcess ( ) ; sr . policy = policy ; return sr ; }


Input: Create PipelineBean which means instantiating all stages for the pipeline . [CODESPLIT] public PipelineBean create ( boolean forExecution , StageLibraryTask library , PipelineConfiguration pipelineConf , InterceptorCreatorContextBuilder interceptorContextBuilder , List < Issue > errors , Map < String , Object > runtimeParameters ) { int priorErrors = errors . size ( ) ; PipelineConfigBean pipelineConfigBean = create ( pipelineConf , errors , runtimeParameters ) ; StageBean errorStageBean = null ; StageBean statsStageBean = null ; StageBean origin = null ; PipelineStageBeans stages = null ; PipelineStageBeans startEventBeans = null ; PipelineStageBeans stopEventBeans = null ; if ( pipelineConfigBean != null && pipelineConfigBean . constants != null ) { Map < String , Object > resolvedConstants = pipelineConfigBean . constants ; if ( interceptorContextBuilder != null ) { interceptorContextBuilder . withExecutionMode ( pipelineConfigBean . executionMode ) . withDeliveryGuarantee ( pipelineConfigBean . deliveryGuarantee ) ; } // Instantiate usual stages if ( ! pipelineConf . getStages ( ) . isEmpty ( ) ) { origin = createStageBean ( forExecution , library , pipelineConf . getStages ( ) . get ( 0 ) , true , false , false , resolvedConstants , interceptorContextBuilder , errors ) ; stages = createPipelineStageBeans ( forExecution , library , pipelineConf . getStages ( ) . subList ( 1 , pipelineConf . getStages ( ) . size ( ) ) , interceptorContextBuilder , resolvedConstants , errors ) ; } // It is not mandatory to have a stats aggregating target configured StageConfiguration statsStageConf = pipelineConf . getStatsAggregatorStage ( ) ; if ( statsStageConf != null ) { statsStageBean = createStageBean ( forExecution , library , statsStageConf , true , false , false , resolvedConstants , interceptorContextBuilder , errors ) ; } // Error stage is mandatory StageConfiguration errorStageConf = pipelineConf . getErrorStage ( ) ; if ( errorStageConf != null ) { errorStageBean = createStageBean ( forExecution , library , errorStageConf , true , true , false , resolvedConstants , interceptorContextBuilder , errors ) ; } else if ( ! ( pipelineConfigBean . executionMode . equals ( ExecutionMode . BATCH ) || pipelineConfigBean . executionMode . equals ( ExecutionMode . STREAMING ) ) ) { errors . add ( IssueCreator . getPipeline ( ) . create ( PipelineGroups . BAD_RECORDS . name ( ) , "badRecordsHandling" , CreationError . CREATION_009 ) ) ; } // Pipeline Lifecycle event handlers StageBean startBean = null ; if ( CollectionUtils . isNotEmpty ( pipelineConf . getStartEventStages ( ) ) ) { startBean = createStageBean ( forExecution , library , pipelineConf . getStartEventStages ( ) . get ( 0 ) , true , false , true , resolvedConstants , interceptorContextBuilder , errors ) ; } startEventBeans = new PipelineStageBeans ( startBean == null ? Collections . emptyList ( ) : ImmutableList . of ( startBean ) ) ; StageBean stopBean = null ; if ( CollectionUtils . isNotEmpty ( pipelineConf . getStopEventStages ( ) ) ) { stopBean = createStageBean ( forExecution , library , pipelineConf . getStopEventStages ( ) . get ( 0 ) , true , false , true , resolvedConstants , interceptorContextBuilder , errors ) ; } stopEventBeans = new PipelineStageBeans ( stopBean == null ? Collections . emptyList ( ) : ImmutableList . of ( stopBean ) ) ; // Validate Webhook Configs if ( pipelineConfigBean . webhookConfigs != null && ! pipelineConfigBean . webhookConfigs . isEmpty ( ) ) { int index = 0 ; for ( PipelineWebhookConfig webhookConfig : pipelineConfigBean . webhookConfigs ) { if ( StringUtils . isEmpty ( webhookConfig . webhookUrl ) ) { Issue issue = IssueCreator . getPipeline ( ) . create ( PipelineGroups . NOTIFICATIONS . name ( ) , "webhookUrl" , CreationError . CREATION_080 ) ; issue . setAdditionalInfo ( "index" , index ) ; errors . add ( issue ) ; break ; } index ++ ; } } } // Something went wrong if ( errors . size ( ) != priorErrors ) { return null ; } return new PipelineBean ( pipelineConfigBean , origin , stages , errorStageBean , statsStageBean , startEventBeans , stopEventBeans ) ; }


Input: Creates additional PipelineStageBeans for additional runners . Stages will share stage definition and thus class loader with the first given runner . That includes stages with private class loader as well . [CODESPLIT] public PipelineStageBeans duplicatePipelineStageBeans ( StageLibraryTask stageLib , PipelineStageBeans pipelineStageBeans , InterceptorCreatorContextBuilder interceptorCreatorContextBuilder , Map < String , Object > constants , List < Issue > errors ) { List < StageBean > stageBeans = new ArrayList <> ( pipelineStageBeans . size ( ) ) ; for ( StageBean original : pipelineStageBeans . getStages ( ) ) { // Create StageDefinition map for this stage Map < Class , ServiceDefinition > services = original . getServices ( ) . stream ( ) . collect ( Collectors . toMap ( c -> c . getDefinition ( ) . getProvides ( ) , ServiceBean :: getDefinition ) ) ; StageBean stageBean = createStage ( stageLib , original . getDefinition ( ) , ClassLoaderReleaser . NOOP_RELEASER , original . getConfiguration ( ) , services :: get , interceptorCreatorContextBuilder , constants , errors ) ; if ( stageBean != null ) { stageBeans . add ( stageBean ) ; } } return new PipelineStageBeans ( stageBeans ) ; }


Input: Create new instance of StageBean . [CODESPLIT] public StageBean createStageBean ( boolean forExecution , StageLibraryTask library , StageConfiguration stageConf , boolean validateAnnotations , boolean errorStage , boolean pipelineLifecycleStage , Map < String , Object > constants , InterceptorCreatorContextBuilder interceptorContextBuilder , List < Issue > errors ) { IssueCreator issueCreator = IssueCreator . getStage ( stageConf . getInstanceName ( ) ) ; StageBean bean = null ; StageDefinition stageDef = library . getStage ( stageConf . getLibrary ( ) , stageConf . getStageName ( ) , forExecution ) ; if ( stageDef != null ) { // Pipeline lifecycle events validation must match, whether it's also marked as error stage does not matter if ( validateAnnotations ) { if ( pipelineLifecycleStage ) { if ( ! stageDef . isPipelineLifecycleStage ( ) ) { errors . add ( issueCreator . create ( CreationError . CREATION_018 , stageDef . getLibraryLabel ( ) , stageDef . getLabel ( ) , stageConf . getStageVersion ( ) ) ) ; } // For non pipeline lifecycle stages, the error stage annotation must match } else if ( stageDef . isErrorStage ( ) != errorStage ) { if ( stageDef . isErrorStage ( ) ) { errors . add ( issueCreator . create ( CreationError . CREATION_007 , stageDef . getLibraryLabel ( ) , stageDef . getLabel ( ) , stageConf . getStageVersion ( ) ) ) ; } else { errors . add ( issueCreator . create ( CreationError . CREATION_008 , stageDef . getLibraryLabel ( ) , stageDef . getLabel ( ) , stageConf . getStageVersion ( ) ) ) ; } } } bean = createStage ( library , stageDef , library , stageConf , serviceClass -> library . getServiceDefinition ( serviceClass , true ) , interceptorContextBuilder , constants , errors ) ; } else { errors . add ( issueCreator . create ( CreationError . CREATION_006 , stageConf . getLibrary ( ) , stageConf . getStageName ( ) , stageConf . getStageVersion ( ) ) ) ; } return bean ; }


Input: Create interceptors for given stage . [CODESPLIT] public List < InterceptorBean > createInterceptors ( StageLibraryTask stageLib , StageConfiguration stageConfiguration , StageDefinition stageDefinition , InterceptorCreatorContextBuilder contextBuilder , InterceptorCreator . InterceptorType interceptorType , List < Issue > issues ) { List < InterceptorBean > beans = new ArrayList <> ( ) ; if ( contextBuilder == null ) { return beans ; } for ( InterceptorDefinition definition : stageLib . getInterceptorDefinitions ( ) ) { InterceptorBean bean = createInterceptor ( stageLib , definition , stageConfiguration , stageDefinition , contextBuilder , interceptorType , issues ) ; if ( bean != null ) { beans . add ( bean ) ; } } return beans ; }


Input: Create a default interceptor for given InterceptorDefinition . This method might return null as the underlying interface for default creation allows it as well - in such case no interceptor is needed . [CODESPLIT] public InterceptorBean createInterceptor ( StageLibraryTask stageLib , InterceptorDefinition definition , StageConfiguration stageConfiguration , StageDefinition stageDefinition , InterceptorCreatorContextBuilder contextBuilder , InterceptorCreator . InterceptorType interceptorType , List < Issue > issues ) { ClassLoader classLoader = Thread . currentThread ( ) . getContextClassLoader ( ) ; InterceptorCreator . Context context = contextBuilder . buildFor ( definition . getLibraryDefinition ( ) . getName ( ) , definition . getKlass ( ) . getName ( ) , stageConfiguration , stageDefinition , interceptorType ) ; try { Thread . currentThread ( ) . setContextClassLoader ( definition . getStageClassLoader ( ) ) ; InterceptorCreator creator = definition . getDefaultCreator ( ) . newInstance ( ) ; Interceptor interceptor = creator . create ( context ) ; if ( interceptor == null ) { return null ; } return new InterceptorBean ( definition , interceptor , stageLib ) ; } catch ( IllegalAccessException | InstantiationException e ) { LOG . debug ( "Can't instantiate interceptor: {}" , e . toString ( ) , e ) ; IssueCreator issueCreator = IssueCreator . getStage ( stageDefinition . getName ( ) ) ; issues . add ( issueCreator . create ( CreationError . CREATION_000 , "interceptor" , definition . getKlass ( ) . getName ( ) , e . toString ( ) ) ) ; } finally { Thread . currentThread ( ) . setContextClassLoader ( classLoader ) ; } return null ; }


Input: Parse JSON representation of Avro schema to Avro s Schema JAVA object [CODESPLIT] public static Schema parseSchema ( String schema ) { Schema . Parser parser = new Schema . Parser ( ) ; parser . setValidate ( true ) ; // We sadly can't use this method directly because it was added after 1.7.3 and we have to stay // compatible with 1.7.3 as this version ships with mapr (and thus we end up using it). This code is // however compiled against 1.7.7 and hence we don't have to do reflection here. try { parser . setValidateDefaults ( true ) ; } catch ( NoSuchMethodError e ) { LOG . debug ( "Running old Avro version that doesn't have 'setValidateDefaults' method" , e ) ; } return parser . parse ( schema ) ; }


Input: Return Date in milliseconds @param days Number of days since unix epoch @return Milliseconds representation for date [CODESPLIT] public static long daysToMillis ( int days ) { long millisUtc = ( long ) days * MILLIS_PER_DAY ; long tmp = millisUtc - ( long ) ( localTimeZone . getOffset ( millisUtc ) ) ; return millisUtc - ( long ) ( localTimeZone . getOffset ( tmp ) ) ; }


Input: Return number of days since the unix epoch . [CODESPLIT] private static int millisToDays ( long millisLocal ) { // We assume millisLocal is midnight of some date. What we are basically trying to do // here is go from local-midnight to UTC-midnight (or whatever time that happens to be). long millisUtc = millisLocal + localTimeZone . getOffset ( millisLocal ) ; int days ; if ( millisUtc >= 0L ) { days = ( int ) ( millisUtc / MILLIS_PER_DAY ) ; } else { days = ( int ) ( ( millisUtc - 86399999 /*(MILLIS_PER_DAY - 1)*/ ) / MILLIS_PER_DAY ) ; } return days ; }


Input: Retrieves avro schema from given header . Throws an exception if the header is missing or is empty . [CODESPLIT] public static String getAvroSchemaFromHeader ( Record record , String headerName ) throws DataGeneratorException { String jsonSchema = record . getHeader ( ) . getAttribute ( headerName ) ; if ( jsonSchema == null || jsonSchema . isEmpty ( ) ) { throw new DataGeneratorException ( Errors . AVRO_GENERATOR_03 , record . getHeader ( ) . getSourceId ( ) ) ; } return jsonSchema ; }


Input: Returns a { @link BigDecimal } from the given bytes and scale . The bytes must adhere to the format that Avro stores decimals in . <br > Avro stores decimal values as two s complement big - endian for the integral portion then the decimal place separately ( via the scale ) . [CODESPLIT] public static BigDecimal bigDecimalFromBytes ( byte [ ] decimalBytes , int scale ) { final BigInteger bigInt = new BigInteger ( decimalBytes ) ; return new BigDecimal ( bigInt , scale ) ; }


Input: Migrating to service for data format library . [CODESPLIT] private void upgradeV1ToV2 ( List < Config > configs , Context context ) { List < Config > dataFormatConfigs = configs . stream ( ) . filter ( c -> c . getName ( ) . startsWith ( "dataFormat" ) ) . collect ( Collectors . toList ( ) ) ; // Remove those configs configs . removeAll ( dataFormatConfigs ) ; // Provide proper prefix dataFormatConfigs = dataFormatConfigs . stream ( ) . map ( c -> new Config ( c . getName ( ) . replace ( "dataFormatConfig." , "dataGeneratorFormatConfig." ) , c . getValue ( ) ) ) . collect ( Collectors . toList ( ) ) ; // And finally register new service context . registerService ( DataFormatGeneratorService . class , dataFormatConfigs ) ; }


Input: buffer size . [CODESPLIT] public long getLength ( ) throws IOException { long length = - 1 ; if ( generator != null ) { length = textOutputStream . getByteCount ( ) ; } else if ( seqWriter != null ) { length = seqWriter . getLength ( ) ; } return length ; }


Input: Copy BlobStore resources to data directory [CODESPLIT] private void copyBlobstore ( List < String > blobStoreResources , File rootDataDir , File pipelineDir ) throws IOException { if ( blobStoreResources == null ) { return ; } File blobstoreDir = new File ( runtimeInfo . getDataDir ( ) , BLOBSTORE_BASE_DIR ) ; File stagingBlobstoreDir = new File ( rootDataDir , BLOBSTORE_BASE_DIR ) ; if ( ! stagingBlobstoreDir . exists ( ) ) { if ( ! stagingBlobstoreDir . mkdirs ( ) ) { throw new RuntimeException ( "Failed to create blobstore directory: " + pipelineDir . getPath ( ) ) ; } } for ( String blobstoreFile : blobStoreResources ) { File srcFile = new File ( blobstoreDir , blobstoreFile ) ; if ( srcFile . exists ( ) ) { final File dstFile = new File ( stagingBlobstoreDir , srcFile . getName ( ) ) ; if ( srcFile . canRead ( ) ) { // ignore files which cannot be read try ( InputStream in = new FileInputStream ( ( srcFile ) ) ) { try ( OutputStream out = new FileOutputStream ( ( dstFile ) ) ) { IOUtils . copy ( in , out ) ; } } } } } }


Input: Reset the listener to use with the next statement . All column information is cleared . [CODESPLIT] public void reset ( ) { columns . clear ( ) ; this . columnsExpected = null ; columnNames = null ; table = null ; schema = null ; insideStatement = false ; }


Input: Updates gauge for the registered thread with the given details . Note that the value of the threadName argument must match the one used to register . [CODESPLIT] public boolean reportHealth ( String threadName , int scheduledDelay , long timestamp ) { ThreadHealthReport threadHealthReport = new ThreadHealthReport ( threadName , scheduledDelay , timestamp ) ; if ( threadToGaugeMap . containsKey ( threadName ) ) { threadToGaugeMap . get ( threadName ) . setThreadHealthReport ( threadHealthReport ) ; return true ; } return false ; }


Input: Creates and registers a Gauge with the given thread name . The same name must be used to report health . [CODESPLIT] public boolean register ( String threadName ) { if ( threadToGaugeMap . containsKey ( threadName ) ) { return false ; } ThreadHealthReportGauge threadHealthReportGauge = new ThreadHealthReportGauge ( ) ; MetricsConfigurator . createGauge ( metrics , getHealthGaugeName ( threadName ) , threadHealthReportGauge , name , rev ) ; threadToGaugeMap . put ( threadName , threadHealthReportGauge ) ; return true ; }


Input: Store configuration from control hub in persistent manner inside data directory . This configuration will be loaded on data collector start and will override any configuration from sdc . properties . [CODESPLIT] public static void storeControlHubConfigs ( RuntimeInfo runtimeInfo , Map < String , String > newConfigs ) throws IOException { File configFile = new File ( runtimeInfo . getDataDir ( ) , SCH_CONF_OVERRIDE ) ; Properties properties = new Properties ( ) ; // Load existing properties from disk if they exists if ( configFile . exists ( ) ) { try ( FileReader reader = new FileReader ( configFile ) ) { properties . load ( reader ) ; } } // Propagate updated configuration for ( Map . Entry < String , String > entry : newConfigs . entrySet ( ) ) { if ( entry . getValue ( ) == null ) { properties . remove ( entry . getKey ( ) ) ; } else { properties . setProperty ( entry . getKey ( ) , entry . getValue ( ) ) ; } } // Store the new updated configuration back to disk try ( FileWriter writer = new FileWriter ( configFile ) ) { properties . store ( writer , null ) ; } }


Input: get the available information about the user <p / > for this LoginModule the credential can be null which will result in a binding ldap authentication scenario <p / > roles are also an optional concept if required [CODESPLIT] @ Override public UserInfo getUserInfo ( String username ) throws Exception { LdapEntry entry = getEntryWithCredential ( username ) ; if ( entry == null ) { return null ; } String pwdCredential = getUserCredential ( entry ) ; pwdCredential = convertCredentialLdapToJetty ( pwdCredential ) ; Credential credential = Credential . getCredential ( pwdCredential ) ; List < String > roles = getUserRoles ( username , entry . getDn ( ) ) ; return new UserInfo ( username , credential , roles ) ; }


Input: attempts to get the users credentials from the users context <p / > NOTE : this is not an user authenticated operation [CODESPLIT] private LdapEntry getEntryWithCredential ( String username ) throws LdapException { if ( StringUtils . isBlank ( _userObjectClass ) || StringUtils . isBlank ( _userIdAttribute ) || StringUtils . isBlank ( _userBaseDn ) || StringUtils . isBlank ( _userPasswordAttribute ) ) { LOG . error ( "Failed to get user because at least one of the following is null : " + "[_userObjectClass, _userIdAttribute, _userBaseDn, _userPasswordAttribute ]" ) ; return null ; } // Create the format of &(objectClass=_userObjectClass)(_userIdAttribute={user})) String userFilter = buildFilter ( _userFilter , _userObjectClass , _userIdAttribute ) ; if ( userFilter . contains ( "{user}" ) ) { userFilter = userFilter . replace ( "{user}" , username ) ; } LOG . debug ( "Searching user using the filter {} on user baseDn {}" , userFilter , _userBaseDn ) ; // Get the group names from each group, which is obtained from roleNameAttribute attribute. SearchRequest request = new SearchRequest ( _userBaseDn , userFilter , _userPasswordAttribute ) ; request . setSearchScope ( SearchScope . SUBTREE ) ; request . setSizeLimit ( 1 ) ; try { SearchOperation search = new SearchOperation ( conn ) ; org . ldaptive . SearchResult result = search . execute ( request ) . getResult ( ) ; LdapEntry entry = result . getEntry ( ) ; LOG . info ( "Found user?: {}" , entry != null ) ; return entry ; } catch ( LdapException ex ) { LOG . error ( "{}" , ex . toString ( ) , ex ) ; return null ; } }


Input: attempts to get the users roles <p / > NOTE : this is not an user authenticated operation [CODESPLIT] private List < String > getUserRoles ( String username , String userDn ) { List < String > roleList = new ArrayList <> ( ) ; if ( StringUtils . isBlank ( _roleBaseDn ) || StringUtils . isBlank ( _roleObjectClass ) || StringUtils . isBlank ( _roleNameAttribute ) || StringUtils . isBlank ( _roleMemberAttribute ) ) { LOG . debug ( "Failed to get roles because at least one of the following is null : " + "[_roleBaseDn, _roleObjectClass, _roleNameAttribute, _roleMemberAttribute ]" ) ; return roleList ; } String roleFilter = buildFilter ( _roleFilter , _roleObjectClass , _roleMemberAttribute ) ; if ( _roleFilter . contains ( DN ) ) { userDn = userDn . replace ( "\\" , "\\\\\\" ) ; roleFilter = roleFilter . replace ( DN , userDn ) ; } else if ( _roleFilter . contains ( USER ) ) { roleFilter = roleFilter . replace ( USER , username ) ; } else { LOG . error ( "roleFilter contains invalid filter {}. Check the roleFilter option" ) ; return roleList ; } LOG . debug ( "Searching roles using the filter {} on role baseDn {}" , roleFilter , _roleBaseDn ) ; // Get the group names from each group, which is obtained from roleNameAttribute attribute. SearchRequest request = new SearchRequest ( _roleBaseDn , roleFilter , _roleNameAttribute ) ; request . setSearchScope ( SearchScope . SUBTREE ) ; try { SearchOperation search = new SearchOperation ( conn ) ; org . ldaptive . SearchResult result = search . execute ( request ) . getResult ( ) ; Collection < LdapEntry > entries = result . getEntries ( ) ; LOG . info ( "Found roles?: {}" , ! ( entries == null || entries . isEmpty ( ) ) ) ; if ( entries != null ) { for ( LdapEntry entry : entries ) { roleList . add ( entry . getAttribute ( ) . getStringValue ( ) ) ; } } } catch ( LdapException ex ) { LOG . error ( ex . getMessage ( ) , ex ) ; } LOG . info ( "Found roles: {}" , roleList ) ; return roleList ; }