Input: Given a filter ( user / role filter ) replace attributes using given information from config . This will create complete filter which will look like & ( objectClass = inetOrgPerson ) ( uid = { user } )) [CODESPLIT] @ VisibleForTesting static String buildFilter ( String attrFilter , String objClass , String attrName ) { // check if the filter has surrounding "()" if ( ! attrFilter . startsWith ( "(" ) ) { attrFilter = "(" + attrFilter ; } if ( ! attrFilter . endsWith ( ")" ) ) { attrFilter = attrFilter + ")" ; } return String . format ( filterFormat , objClass , String . format ( attrFilter , attrName ) ) ; }


Input: since ldap uses a context bind for valid authentication checking we override login () <p / > if credentials are not available from the users context or if we are forcing the binding check then we try a binding authentication check otherwise if we have the users encoded password then we can try authentication via that mechanic [CODESPLIT] @ Override public boolean login ( ) throws LoginException { try { if ( getCallbackHandler ( ) == null ) { throw new LoginException ( "No callback handler" ) ; } if ( conn == null ) { return false ; } Callback [ ] callbacks = configureCallbacks ( ) ; getCallbackHandler ( ) . handle ( callbacks ) ; String webUserName = ( ( NameCallback ) callbacks [ 0 ] ) . getName ( ) ; Object webCredential = ( ( ObjectCallback ) callbacks [ 1 ] ) . getObject ( ) ; if ( webUserName == null || webCredential == null ) { setAuthenticated ( false ) ; return isAuthenticated ( ) ; } // Please see the following stackoverflow article // http://security.stackexchange.com/questions/6713/ldap-security-problems // Some LDAP implementation "MAY" accept empty password as a sign of anonymous connection and thus // return "true" for the authentication request. if ( ( webCredential instanceof String ) && ( ( String ) webCredential ) . isEmpty ( ) ) { LOG . info ( "Ignoring login request for user {} as the password is empty." , webUserName ) ; setAuthenticated ( false ) ; return isAuthenticated ( ) ; } if ( _forceBindingLogin ) { return bindingLogin ( webUserName , webCredential ) ; } // This sets read and the credential UserInfo userInfo = getUserInfo ( webUserName ) ; if ( userInfo == null ) { setAuthenticated ( false ) ; return false ; } JAASUserInfo jaasUserInfo = new JAASUserInfo ( userInfo ) ; jaasUserInfo . fetchRoles ( ) ; setCurrentUser ( jaasUserInfo ) ; if ( webCredential instanceof String ) { return credentialLogin ( Credential . getCredential ( ( String ) webCredential ) ) ; } return credentialLogin ( webCredential ) ; } catch ( UnsupportedCallbackException e ) { throw new LoginException ( "Error obtaining callback information." ) ; } catch ( IOException e ) { LOG . error ( "IO Error performing login" , e ) ; } catch ( Exception e ) { LOG . error ( "IO Error performing login" , e ) ; } return false ; }


Input: password supplied authentication check [CODESPLIT] protected boolean credentialLogin ( Object webCredential ) throws LoginException { boolean credResult = getCurrentUser ( ) . checkCredential ( webCredential ) ; setAuthenticated ( credResult ) ; if ( ! credResult ) { LOG . warn ( "Authentication failed - Possibly the user password is wrong" ) ; } return isAuthenticated ( ) ; }


Input: binding authentication check This method of authentication works only if the user branch of the DIT ( ldap tree ) has an ACI ( access control instruction ) that allow the access to any user or at least for the user that logs in . [CODESPLIT] public boolean bindingLogin ( String username , Object password ) throws Exception { if ( StringUtils . isBlank ( _userObjectClass ) || StringUtils . isBlank ( _userIdAttribute ) || StringUtils . isBlank ( _userBaseDn ) ) { LOG . error ( "Failed to get user because at least one of the following is null : " + "[_userObjectClass, _userIdAttribute, _userBaseDn ]" ) ; return false ; } LdapEntry userEntry = authenticate ( username , password ) ; if ( userEntry == null ) { return false ; } // If authenticated by LDAP server, the returned LdapEntry contains full DN of the user String userDn = userEntry . getDn ( ) ; if ( userDn == null ) { // This shouldn't happen if LDAP server is configured properly. LOG . error ( "userDn is found null for the user {}" , username ) ; return false ; } List < String > roles = getUserRoles ( username , userDn ) ; //Authentication already succeeded. We won't store user password so passing empty credential UserInfo userInfo = new UserInfo ( username , Credential . getCredential ( "" ) , roles ) ; JAASUserInfo jaasUserInfo = new JAASUserInfo ( userInfo ) ; jaasUserInfo . fetchRoles ( ) ; setCurrentUser ( jaasUserInfo ) ; setAuthenticated ( true ) ; return true ; }


Input: Perform authentication with given username and password . Receive the result from Ldap server [CODESPLIT] private LdapEntry authenticate ( String username , Object password ) { try { SearchDnResolver dnResolver = new SearchDnResolver ( new DefaultConnectionFactory ( connConfig ) ) ; dnResolver . setBaseDn ( _userBaseDn ) ; dnResolver . setSubtreeSearch ( true ) ; String userFilter = buildFilter ( _userFilter , _userObjectClass , _userIdAttribute ) ; LOG . debug ( "Searching a user with filter {} where user is {}" , userFilter , username ) ; dnResolver . setUserFilter ( userFilter ) ; // Set Authenticator with username and password. It will return the user if username/password matches. BindAuthenticationHandler authHandler = new BindAuthenticationHandler ( new DefaultConnectionFactory ( connConfig ) ) ; Authenticator auth = new Authenticator ( dnResolver , authHandler ) ; AuthenticationRequest authRequest = new AuthenticationRequest ( ) ; authRequest . setUser ( username ) ; if ( password instanceof char [ ] ) { authRequest . setCredential ( new org . ldaptive . Credential ( new String ( ( char [ ] ) password ) ) ) ; } else if ( password instanceof String ) { authRequest . setCredential ( new org . ldaptive . Credential ( ( String ) password ) ) ; } else { LOG . error ( "Unexpected type for password '{}'" , ( password != null ) ? password . getClass ( ) : "NULL" ) ; return null ; } String [ ] userRoleAttribute = ReturnAttributes . ALL . value ( ) ; authRequest . setReturnAttributes ( userRoleAttribute ) ; LOG . debug ( "Retrieved authenticator from factory: {}" , auth ) ; LOG . debug ( "Retrieved authentication request from factory: {}" , authRequest ) ; AuthenticationResponse response = auth . authenticate ( authRequest ) ; LOG . info ( "Found user?: {}" , response . getResult ( ) ) ; if ( response . getResult ( ) ) { LdapEntry entry = response . getLdapEntry ( ) ; return entry ; } else { // User not found. Most likely username/password didn't match. Log the reason. LOG . error ( "Result code: {} - {}" , response . getResultCode ( ) , response . getMessage ( ) ) ; } }


Input: <p > Move a series of config values from old names to new names ( one to one correspondence ) . Config values will be preserved . < / p > [CODESPLIT] public static int moveAllTo ( List < Config > configs , String ... names ) { Map < String , String > nameMap = new HashMap <> ( ) ; if ( names . length % 2 == 1 ) { throw new IllegalArgumentException ( "names was of uneven length" ) ; } for ( int i = 0 ; i < names . length ; ) { nameMap . put ( names [ i ] , names [ i + 1 ] ) ; i += 2 ; } return moveAllTo ( configs , nameMap ) ; }


Input: <p > Move a series of config values from old names to new names ( one to one correspondence ) . Config values will be preserved . < / p > [CODESPLIT] public static int moveAllTo ( List < Config > configs , Map < String , String > oldToNewNames ) { List < Config > configsToAdd = new ArrayList <> ( ) ; List < Config > configsToRemove = new ArrayList <> ( ) ; int numMoved = 0 ; for ( Config config : configs ) { final String oldName = config . getName ( ) ; if ( oldToNewNames . containsKey ( oldName ) ) { configsToRemove . add ( config ) ; final Object value = config . getValue ( ) ; final String newName = oldToNewNames . get ( oldName ) ; configsToAdd . add ( new Config ( newName , value ) ) ; LOG . info ( String . format ( "Moving config value %s from old name %s to new name %s" , value , oldName , newName ) ) ; numMoved ++ ; } } configs . removeAll ( configsToRemove ) ; configs . addAll ( configsToAdd ) ; return numMoved ; }


Input: <p > Returns a { @link Config } object from the supplied list with the supplied name if it exists . If a non - null Config is returned the supplied list of { @code configs } will be modified such that it no longer contains the returned value . < / p > [CODESPLIT] public static Config getAndRemoveConfigWithName ( List < Config > configs , String name ) { final Config config = getConfigWithName ( configs , name ) ; if ( config != null ) { configs . remove ( config ) ; } return config ; }


Input: Re - login a principal . This method assumes that { [CODESPLIT] private synchronized void reLogin ( ) throws LoginException { if ( ! isKrbTicket ) { return ; } if ( login == null ) { throw new LoginException ( "Login must be done first" ) ; } if ( ! hasSufficientTimeElapsed ( ) ) { return ; } log . info ( "Initiating logout for {}" , principal ) ; synchronized ( Login . class ) { // register most recent relogin attempt lastLogin = currentElapsedTime ( ) ; //clear up the kerberos state. But the tokens are not cleared! As per //the Java kerberos login module code, only the kerberos credentials //are cleared login . logout ( ) ; //login and also update the subject field of this instance to //have the new credentials (pass it to the LoginContext constructor) login = new LoginContext ( loginContextName , subject ) ; log . info ( "Initiating re-login for {}" , principal ) ; login . login ( ) ; } }


Input: Upgrade whole pipeline at once and return updated variant . [CODESPLIT] public PipelineConfiguration upgradeIfNecessary ( StageLibraryTask library , PipelineConfiguration pipelineConf , List < Issue > issues ) { Preconditions . checkArgument ( issues . isEmpty ( ) , "Given list of issues must be empty." ) ; boolean upgrade ; // Firstly upgrading schema if needed, then data upgrade = needsSchemaUpgrade ( pipelineConf , issues ) ; if ( upgrade && issues . isEmpty ( ) ) { pipelineConf = upgradeSchema ( library , pipelineConf , issues ) ; } // Something went wrong with the schema upgrade if ( ! issues . isEmpty ( ) ) { return null ; } // Upgrading data if needed upgrade = needsUpgrade ( library , pipelineConf , issues ) ; if ( upgrade && issues . isEmpty ( ) ) { //we try to upgrade only if we have all defs for the pipelineConf pipelineConf = upgrade ( library , pipelineConf , issues ) ; } return ( issues . isEmpty ( ) ) ? pipelineConf : null ; }


Input: Upgrade detached stage ( stage not associated directly with the pipeline ) . [CODESPLIT] public StageConfiguration upgradeIfNecessary ( StageLibraryTask libraryTask , StageConfiguration stageConf , List < Issue > issues ) { Preconditions . checkArgument ( issues . isEmpty ( ) , "Given list of issues must be empty." ) ; boolean upgrade = needsUpgrade ( libraryTask , stageConf , issues ) ; if ( upgrade ) { stageConf = upgradeIfNeeded ( libraryTask , stageConf , issues ) ; } return issues . isEmpty ( ) ? stageConf : null ; }


Input: Upgrade whole Stage configuration including all services if needed . Convenience method that will lookup stage definition from the library . [CODESPLIT] static StageConfiguration upgradeIfNeeded ( StageLibraryTask library , StageConfiguration conf , List < Issue > issues ) { return upgradeIfNeeded ( library , library . getStage ( conf . getLibrary ( ) , conf . getStageName ( ) , false ) , conf , issues ) ; }


Input: Upgrade whole Stage configuration including all services if needed . [CODESPLIT] static StageConfiguration upgradeIfNeeded ( StageLibraryTask library , StageDefinition def , StageConfiguration conf , List < Issue > issues ) { IssueCreator issueCreator = IssueCreator . getStage ( conf . getInstanceName ( ) ) ; int fromVersion = conf . getStageVersion ( ) ; int toVersion = def . getVersion ( ) ; try { // Firstly upgrade stage itself (register any new services) upgradeStageIfNeeded ( def , conf , issueCreator , issues ) ; // And then upgrade all it's services conf . getServices ( ) . forEach ( serviceConf -> upgradeServicesIfNeeded ( library , conf , serviceConf , issueCreator . forService ( serviceConf . getService ( ) . getName ( ) ) , issues ) ) ; } catch ( Exception ex ) { LOG . error ( "Unknown exception during upgrade: " + ex , ex ) ; issues . add ( issueCreator . create ( ContainerError . CONTAINER_0900 , fromVersion , toVersion , ex . toString ( ) ) ) ; } return conf ; }


Input: Internal method that will upgrade service configuration if needed . [CODESPLIT] private static ServiceConfiguration upgradeServicesIfNeeded ( StageLibraryTask library , StageConfiguration stageConf , ServiceConfiguration conf , IssueCreator issueCreator , List < Issue > issues ) { ServiceDefinition def = library . getServiceDefinition ( conf . getService ( ) , false ) ; if ( def == null ) { issues . add ( issueCreator . create ( ContainerError . CONTAINER_0903 , conf . getService ( ) . getName ( ) ) ) ; } int fromVersion = conf . getServiceVersion ( ) ; int toVersion = def . getVersion ( ) ; // In case we don't need an upgrade if ( ! needsUpgrade ( toVersion , fromVersion , issueCreator , issues ) ) { return conf ; } ClassLoader cl = Thread . currentThread ( ) . getContextClassLoader ( ) ; try { LOG . warn ( "Upgrading service instance from version '{}' to version '{}'" , conf . getServiceVersion ( ) , def . getVersion ( ) ) ; UpgradeContext upgradeContext = new UpgradeContext ( "" , def . getName ( ) , stageConf . getInstanceName ( ) , fromVersion , toVersion ) ; List < Config > configs = def . getUpgrader ( ) . upgrade ( conf . getConfiguration ( ) , upgradeContext ) ; if ( ! upgradeContext . registeredServices . isEmpty ( ) ) { throw new StageException ( ContainerError . CONTAINER_0904 ) ; } conf . setServiceVersion ( toVersion ) ; conf . setConfig ( configs ) ; } catch ( StageException ex ) { issues . add ( issueCreator . create ( ex . getErrorCode ( ) , ex . getParams ( ) ) ) ; } finally { Thread . currentThread ( ) . setContextClassLoader ( cl ) ; } return conf ; }


Input: Internal method that will upgrade only Stage configuration - not the associated services - and only if needed . [CODESPLIT] static private void upgradeStageIfNeeded ( StageDefinition def , StageConfiguration conf , IssueCreator issueCreator , List < Issue > issues ) { int fromVersion = conf . getStageVersion ( ) ; int toVersion = def . getVersion ( ) ; // In case we don't need an upgrade if ( ! needsUpgrade ( toVersion , fromVersion , IssueCreator . getStage ( conf . getInstanceName ( ) ) , issues ) ) { return ; } ClassLoader cl = Thread . currentThread ( ) . getContextClassLoader ( ) ; try { Thread . currentThread ( ) . setContextClassLoader ( def . getStageClassLoader ( ) ) ; LOG . warn ( "Upgrading stage instance '{}' from version '{}' to version '{}'" , conf . getInstanceName ( ) , fromVersion , toVersion ) ; UpgradeContext upgradeContext = new UpgradeContext ( def . getLibrary ( ) , def . getName ( ) , conf . getInstanceName ( ) , fromVersion , toVersion ) ; List < Config > configs = def . getUpgrader ( ) . upgrade ( conf . getConfiguration ( ) , upgradeContext ) ; conf . setStageVersion ( def . getVersion ( ) ) ; conf . setConfig ( configs ) ; // Propagate newly registered services to the StageConfiguration if ( ! upgradeContext . registeredServices . isEmpty ( ) ) { List < ServiceConfiguration > services = new ArrayList <> ( ) ; services . addAll ( conf . getServices ( ) ) ; // Version -1 is special to note that this version has been created by stage and not by the service itself upgradeContext . registeredServices . forEach ( ( s , c ) -> services . add ( new ServiceConfiguration ( s , - 1 , c ) ) ) ; conf . setServices ( services ) ; } } catch ( StageException ex ) { issues . add ( issueCreator . create ( ex . getErrorCode ( ) , ex . getParams ( ) ) ) ; } finally { Thread . currentThread ( ) . setContextClassLoader ( cl ) ; } }


Input: Sets the file offsets to use for the next read . To work correctly the last return offsets should be used or an empty <code > Map< / code > if there is none . <p / > If a reader is already live the corresponding set offset is ignored as we cache all the contextual information of live readers . [CODESPLIT] @ Override public void setOffsets ( Map < String , String > offsets ) throws IOException { Utils . checkNotNull ( offsets , "offsets" ) ; LOG . trace ( "setOffsets()" ) ; // We look for created directory paths here findCreatedDirectories ( ) ; // we look for new files only here findNewFileContexts ( ) ; // we purge file only here purge ( ) ; super . setOffsets ( offsets ) ; startNewLoop ( ) ; }


Input: should be replaced by null . [CODESPLIT] private List < String > getFieldsToNull ( List < NullReplacerConditionalConfig > nullReplacerConditionalConfigs , Set < String > fieldsThatDoNotExist , Set < String > fieldPaths , Record record ) throws OnRecordErrorException { //Gather in this all fields to null List < String > fieldsToNull = new ArrayList <> ( ) ; for ( NullReplacerConditionalConfig nullReplacerConditionalConfig : nullReplacerConditionalConfigs ) { List < String > fieldNamesToNull = nullReplacerConditionalConfig . fieldsToNull ; //Gather fieldsPathsToNull for this nullReplacerConditionalConfig List < String > fieldPathsToNull = new ArrayList <> ( ) ; //Gather existing paths for each nullReplacerConditionalConfig //And if field does not exist gather them in fieldsThatDoNotExist for ( String fieldNameToNull : fieldNamesToNull ) { try { final List < String > matchingPaths = FieldPathExpressionUtil . evaluateMatchingFieldPaths ( fieldNameToNull , fieldPathEval , fieldPathVars , record , fieldPaths ) ; if ( matchingPaths . isEmpty ( ) ) { // FieldPathExpressionUtil.evaluateMatchingFieldPaths does NOT return the supplied param in its result // regardless, like FieldRegexUtil#getMatchingFieldPaths did, so we add manually here fieldsThatDoNotExist . add ( fieldNameToNull ) ; } else { for ( String matchingField : matchingPaths ) { if ( record . has ( matchingField ) ) { fieldPathsToNull . add ( matchingField ) ; } else { fieldsThatDoNotExist . add ( matchingField ) ; } } } } catch ( ELEvalException e ) { LOG . error ( "Error evaluating condition: " + nullReplacerConditionalConfig . condition , e ) ; throw new OnRecordErrorException ( record , Errors . VALUE_REPLACER_07 , fieldNameToNull , e . toString ( ) , e ) ; } } //Now evaluate the condition in nullReplacerConditionalConfig //If it empty or condition evaluates to true, add all the gathered fields in fieldsPathsToNull // for this nullReplacerConditionalConfig to fieldsToNull try { boolean evaluatedCondition = true ; //If it is empty we assume it is true. if ( ! StringUtils . isEmpty ( nullReplacerConditionalConfig . condition ) ) { evaluatedCondition = nullConditionELEval . eval ( nullConditionELVars , nullReplacerConditionalConfig . condition , Boolean . class ) ; } if ( evaluatedCondition ) { fieldsToNull . addAll ( fieldPathsToNull ) ; } } catch ( ELEvalException e ) { LOG . error ( "Error evaluating condition: " + nullReplacerConditionalConfig . condition , e ) ; throw new OnRecordErrorException ( record , Errors . VALUE_REPLACER_06 , nullReplacerConditionalConfig . condition , e . toString ( ) ) ; } } return fieldsToNull ; }


Input: Checks if a class should be included as a system class . [CODESPLIT] private static boolean isSystemClass ( String name , List < String > packageList ) { boolean result = false ; if ( packageList != null ) { String canonicalName = ClassLoaderUtil . canonicalizeClassOrResource ( name ) ; for ( String c : packageList ) { boolean shouldInclude = true ; if ( c . startsWith ( "-" ) ) { c = c . substring ( 1 ) ; shouldInclude = false ; } if ( canonicalName . startsWith ( c ) ) { if ( c . endsWith ( "." ) // package || canonicalName . length ( ) == c . length ( ) // class || canonicalName . length ( ) > c . length ( ) // nested && canonicalName . charAt ( c . length ( ) ) == ' ' ) { if ( shouldInclude ) { result = true ; } else { return false ; } } } } } return result ; }


Input: Visible For Testing ( TODO - Import guava and make sure it doesn t conflict with spark s rootclassloader ) [CODESPLIT] CheckpointPath getCheckPointPath ( String topic , String consumerGroup ) { return new CheckpointPath . Builder ( CHECKPOINT_BASE_DIR ) . sdcId ( Utils . getPropertyNotNull ( properties , SDC_ID ) ) . topic ( topic ) . consumerGroup ( consumerGroup ) . pipelineName ( Utils . getPropertyNotNull ( properties , ClusterModeConstants . CLUSTER_PIPELINE_NAME ) ) . build ( ) ; }


Input: Adds the given element to this queue . If the queue is currently full the element at the head of the queue is evicted to make room . [CODESPLIT] @ Override public boolean add ( E e ) { checkNotNull ( e ) ; // check before removing if ( maxSize == 0 ) { return true ; } if ( size ( ) == maxSize ) { delegate . remove ( ) ; } delegate . add ( e ) ; return true ; }


Input: Adds the given element to this queue . If the queue is currently full the element at the head of the queue is evicted to make room and returns the evicted element . [CODESPLIT] public E addAndGetEvicted ( E e ) { checkNotNull ( e ) ; // check before removing if ( maxSize == 0 ) { return null ; } E evicted = null ; if ( size ( ) == maxSize ) { evicted = delegate . remove ( ) ; } delegate . add ( e ) ; return evicted ; }


Input: Internal method to ensure that we return and not cache the default value if needed [CODESPLIT] private Optional < Value > valueOrDefault ( Key key , Optional < Value > value ) { // If value is present simply return it if ( value . isPresent ( ) ) { return value ; } if ( ! cacheMissingValues ) { delegate . invalidate ( key ) ; } return defaultValue ; }


Input: Preview only returns data associated with batches however errors are reported outside of batch context for multi - threaded pipelines . Thus we emulate the behavior by simply adding into the current batch all so - far reported errors . [CODESPLIT] private List < StageOutput > addReportedErrorsIfNeeded ( List < StageOutput > snapshotsOfAllStagesOutput ) { synchronized ( this . reportedErrors ) { if ( reportedErrors . isEmpty ( ) ) { return snapshotsOfAllStagesOutput ; } try { return snapshotsOfAllStagesOutput . stream ( ) . map ( so -> new StageOutput ( so . getInstanceName ( ) , so . getOutput ( ) , so . getErrorRecords ( ) , reportedErrors . get ( so . getInstanceName ( ) ) , so . getEventRecords ( ) ) ) . collect ( Collectors . toList ( ) ) ; } finally { reportedErrors . clear ( ) ; } } }


Input: Returns a protobuf descriptor instance from the provided descriptor file . [CODESPLIT] public static Descriptors . Descriptor getDescriptor ( ProtoConfigurableEntity . Context context , String protoDescriptorFile , String messageType , Map < String , Set < Descriptors . FieldDescriptor > > messageTypeToExtensionMap , Map < String , Object > defaultValueMap ) throws StageException { File descriptorFileHandle = new File ( context . getResourcesDirectory ( ) , protoDescriptorFile ) ; try ( FileInputStream fin = new FileInputStream ( descriptorFileHandle ) ; ) { DescriptorProtos . FileDescriptorSet set = DescriptorProtos . FileDescriptorSet . parseFrom ( fin ) ; // Iterate over all the file descriptor set computed above and cache dependencies and all encountered // file descriptors // this map holds all the dependencies that a given file descriptor has. // This cached map will be looked up while building FileDescriptor instances Map < String , Set < Descriptors . FileDescriptor > > fileDescriptorDependentsMap = new HashMap <> ( ) ; // All encountered FileDescriptor instances cached based on their name. Map < String , Descriptors . FileDescriptor > fileDescriptorMap = new HashMap <> ( ) ; ProtobufTypeUtil . getAllFileDescriptors ( set , fileDescriptorDependentsMap , fileDescriptorMap ) ; // Get the descriptor for the expected message type Descriptors . Descriptor descriptor = ProtobufTypeUtil . getDescriptor ( set , fileDescriptorMap , protoDescriptorFile , messageType ) ; // Compute and cache all extensions defined for each message type ProtobufTypeUtil . populateDefaultsAndExtensions ( fileDescriptorMap , messageTypeToExtensionMap , defaultValueMap ) ; return descriptor ; } catch ( FileNotFoundException e ) { throw new StageException ( Errors . PROTOBUF_06 , descriptorFileHandle . getAbsolutePath ( ) , e ) ; } catch ( IOException e ) { throw new StageException ( Errors . PROTOBUF_08 , e . toString ( ) , e ) ; } }


Input: Loads a Protobuf file descriptor set into an ubermap of file descriptors . [CODESPLIT] public static void getAllFileDescriptors ( DescriptorProtos . FileDescriptorSet set , Map < String , Set < Descriptors . FileDescriptor > > dependenciesMap , Map < String , Descriptors . FileDescriptor > fileDescriptorMap ) throws StageException { List < DescriptorProtos . FileDescriptorProto > fileList = set . getFileList ( ) ; try { for ( DescriptorProtos . FileDescriptorProto fdp : fileList ) { if ( ! fileDescriptorMap . containsKey ( fdp . getName ( ) ) ) { Set < Descriptors . FileDescriptor > dependencies = dependenciesMap . get ( fdp . getName ( ) ) ; if ( dependencies == null ) { dependencies = new LinkedHashSet <> ( ) ; dependenciesMap . put ( fdp . getName ( ) , dependencies ) ; dependencies . addAll ( getDependencies ( dependenciesMap , fileDescriptorMap , fdp , set ) ) ; } Descriptors . FileDescriptor fileDescriptor = Descriptors . FileDescriptor . buildFrom ( fdp , dependencies . toArray ( new Descriptors . FileDescriptor [ dependencies . size ( ) ] ) ) ; fileDescriptorMap . put ( fdp . getName ( ) , fileDescriptor ) ; } } } catch ( Descriptors . DescriptorValidationException e ) { throw new StageException ( Errors . PROTOBUF_07 , e . getDescription ( ) , e ) ; } }


Input: Populates a map of protobuf extensions and map with the default values for each message field from a map of file descriptors . [CODESPLIT] public static void populateDefaultsAndExtensions ( Map < String , Descriptors . FileDescriptor > fileDescriptorMap , Map < String , Set < Descriptors . FieldDescriptor > > typeToExtensionMap , Map < String , Object > defaultValueMap ) { for ( Descriptors . FileDescriptor f : fileDescriptorMap . values ( ) ) { // go over every file descriptor and look for extensions and default values of those extensions for ( Descriptors . FieldDescriptor fieldDescriptor : f . getExtensions ( ) ) { String containingType = fieldDescriptor . getContainingType ( ) . getFullName ( ) ; Set < Descriptors . FieldDescriptor > fieldDescriptors = typeToExtensionMap . get ( containingType ) ; if ( fieldDescriptors == null ) { fieldDescriptors = new LinkedHashSet <> ( ) ; typeToExtensionMap . put ( containingType , fieldDescriptors ) ; } fieldDescriptors . add ( fieldDescriptor ) ; if ( fieldDescriptor . hasDefaultValue ( ) ) { defaultValueMap . put ( containingType + "." + fieldDescriptor . getName ( ) , fieldDescriptor . getDefaultValue ( ) ) ; } } // go over messages within file descriptor and look for all fields and extensions and their defaults for ( Descriptors . Descriptor d : f . getMessageTypes ( ) ) { addDefaultsAndExtensions ( typeToExtensionMap , defaultValueMap , d ) ; } } }


Input: Generates a protobuf descriptor instance from a FileDescriptor set . [CODESPLIT] public static Descriptors . Descriptor getDescriptor ( DescriptorProtos . FileDescriptorSet set , Map < String , Descriptors . FileDescriptor > fileDescriptorMap , String descriptorFile , String qualifiedMessageType ) throws StageException { // find the FileDescriptorProto which contains the message type // IF cannot find, then bail out String packageName = null ; String messageType = qualifiedMessageType ; int lastIndex = qualifiedMessageType . lastIndexOf ( ' ' ) ; if ( lastIndex != - 1 ) { packageName = qualifiedMessageType . substring ( 0 , lastIndex ) ; messageType = qualifiedMessageType . substring ( lastIndex + 1 ) ; } DescriptorProtos . FileDescriptorProto file = getFileDescProtoForMsgType ( packageName , messageType , set ) ; if ( file == null ) { // could not find the message type from all the proto files contained in the descriptor file throw new StageException ( Errors . PROTOBUF_00 , qualifiedMessageType , descriptorFile ) ; } // finally get the FileDescriptor for the message type Descriptors . FileDescriptor fileDescriptor = fileDescriptorMap . get ( file . getName ( ) ) ; // create builder using the FileDescriptor // this can only find the top level message types return fileDescriptor . findMessageTypeByName ( messageType ) ; }


Input: Converts a protobuf message to an SDC Record Field . [CODESPLIT] public static Field protobufToSdcField ( Record record , String fieldPath , Descriptors . Descriptor descriptor , Map < String , Set < Descriptors . FieldDescriptor > > messageTypeToExtensionMap , Object message ) throws DataParserException { LinkedHashMap < String , Field > sdcRecordMapFieldValue = new LinkedHashMap <> ( ) ; // get all the expected fields from the proto file Map < String , Descriptors . FieldDescriptor > protobufFields = new LinkedHashMap <> ( ) ; for ( Descriptors . FieldDescriptor fieldDescriptor : descriptor . getFields ( ) ) { protobufFields . put ( fieldDescriptor . getName ( ) , fieldDescriptor ) ; } // get all fields in the read message Map < Descriptors . FieldDescriptor , Object > values = ( ( DynamicMessage ) message ) . getAllFields ( ) ; // for every field present in the proto definition create an sdc field. for ( Descriptors . FieldDescriptor fieldDescriptor : protobufFields . values ( ) ) { Object value = values . get ( fieldDescriptor ) ; sdcRecordMapFieldValue . put ( fieldDescriptor . getName ( ) , createField ( record , fieldPath , fieldDescriptor , messageTypeToExtensionMap , value ) ) ; } // handle applicable extensions for this message type if ( messageTypeToExtensionMap . containsKey ( descriptor . getFullName ( ) ) ) { for ( Descriptors . FieldDescriptor fieldDescriptor : messageTypeToExtensionMap . get ( descriptor . getFullName ( ) ) ) { if ( values . containsKey ( fieldDescriptor ) ) { Object value = values . get ( fieldDescriptor ) ; sdcRecordMapFieldValue . put ( fieldDescriptor . getName ( ) , createField ( record , fieldPath , fieldDescriptor , messageTypeToExtensionMap , value ) ) ; } } } // handle unknown fields // unknown fields can go into the record header UnknownFieldSet unknownFields = ( ( DynamicMessage ) message ) . getUnknownFields ( ) ; if ( ! unknownFields . asMap ( ) . isEmpty ( ) ) { ByteArrayOutputStream bOut = new ByteArrayOutputStream ( ) ; try { unknownFields . writeDelimitedTo ( bOut ) ; bOut . flush ( ) ; bOut . close ( ) ; } catch ( IOException e ) { throw new DataParserException ( Errors . PROTOBUF_10 , e . toString ( ) , e ) ; } String path = fieldPath . isEmpty ( ) ? FORWARD_SLASH : fieldPath ; byte [ ] bytes = org . apache . commons . codec . binary . Base64 . encodeBase64 ( bOut . toByteArray ( ) ) ; record . getHeader ( ) . setAttribute ( PROTOBUF_UNKNOWN_FIELDS_PREFIX + path , new String ( bytes , StandardCharsets . UTF_8 ) ) ; } return Field . createListMap ( sdcRecordMapFieldValue ) ; }


Input: Creates an SDC Record Field from the provided protobuf message and descriptor . [CODESPLIT] @ SuppressWarnings ( "unchecked" ) private static Field createField ( Record record , String fieldPath , Descriptors . FieldDescriptor fieldDescriptor , Map < String , Set < Descriptors . FieldDescriptor > > messageTypeToExtensionMap , Object message ) throws DataParserException { Field newField ; if ( message == null ) { // If the message does not contain required fields then builder.build() throws UninitializedMessageException Object defaultValue = null ; Descriptors . FieldDescriptor . JavaType javaType = fieldDescriptor . getJavaType ( ) ; // get default values only for optional fields and non-message types if ( fieldDescriptor . isOptional ( ) && fieldDescriptor . getJavaType ( ) != Descriptors . FieldDescriptor . JavaType . MESSAGE ) { defaultValue = fieldDescriptor . getDefaultValue ( ) ; //Default value for byte string should be converted to byte array if ( javaType == Descriptors . FieldDescriptor . JavaType . BYTE_STRING && defaultValue instanceof ByteString ) { defaultValue = ( ( ByteString ) defaultValue ) . toByteArray ( ) ; } } newField = Field . create ( getFieldType ( javaType ) , defaultValue ) ; } else if ( fieldDescriptor . isMapField ( ) ) { // Map entry (protobuf 3 map) Map < String , Field > sdcMapFieldValues = new HashMap <> ( ) ; Collection < DynamicMessage > mapEntries = ( Collection < DynamicMessage > ) message ; // MapEntry for ( DynamicMessage dynamicMessage : mapEntries ) { // MapEntry has 2 fields, key and value Map < Descriptors . FieldDescriptor , Object > kv = dynamicMessage . getAllFields ( ) ; String key = null ; Object value = null ; Descriptors . FieldDescriptor valueDescriptor = null ; for ( Map . Entry < Descriptors . FieldDescriptor , Object > entry : kv . entrySet ( ) ) { switch ( entry . getKey ( ) . getName ( ) ) { case KEY : key = entry . getValue ( ) . toString ( ) ; break ; case VALUE : value = entry . getValue ( ) ; valueDescriptor = entry . getKey ( ) ; break ; default : throw new DataParserException ( Errors . PROTOBUF_09 , entry . getKey ( ) . getName ( ) ) ; } } if ( key != null && valueDescriptor != null ) { sdcMapFieldValues . put ( key , createSdcField ( record , fieldPath , valueDescriptor , messageTypeToExtensionMap , value ) ) ; } } newField = Field . create ( sdcMapFieldValues ) ; } else if ( fieldDescriptor . isRepeated ( ) ) { // List entry (repeated) List < ? > list = ( List < ? > ) message ; List < Field > listField = new ArrayList <> ( ) ; for ( int i = 0 ; i < list . size ( ) ; i ++ ) { if ( fieldDescriptor . getJavaType ( ) == Descriptors . FieldDescriptor . JavaType . MESSAGE ) { listField . add ( protobufToSdcField ( record , fieldPath + "[" + i + "]" , fieldDescriptor . getMessageType ( ) , messageTypeToExtensionMap , list . get ( i ) ) ) ; } else { listField . add ( createSdcField ( record , fieldPath + "[" + i + "]" , fieldDescriptor , messageTypeToExtensionMap , list . get ( i ) ) ) ; } } newField = Field . create ( listField ) ; } else { // normal entry newField = createSdcField ( record , fieldPath , fieldDescriptor , messageTypeToExtensionMap , message ) ; } return newField ; }


Input: Serializes a record to a protobuf message using the specified descriptor . [CODESPLIT] public static DynamicMessage sdcFieldToProtobufMsg ( Record record , Descriptors . Descriptor desc , Map < String , Set < Descriptors . FieldDescriptor > > messageTypeToExtensionMap , Map < String , Object > defaultValueMap ) throws DataGeneratorException { return sdcFieldToProtobufMsg ( record , record . get ( ) , "" , desc , messageTypeToExtensionMap , defaultValueMap ) ; }


Input: Serializes a field path in a record to a protobuf message using the specified descriptor . [CODESPLIT] private static DynamicMessage sdcFieldToProtobufMsg ( Record record , Field field , String fieldPath , Descriptors . Descriptor desc , Map < String , Set < Descriptors . FieldDescriptor > > messageTypeToExtensionMap , Map < String , Object > defaultValueMap ) throws DataGeneratorException { if ( field == null ) { return null ; } // compute all fields to look for including extensions DynamicMessage . Builder builder = DynamicMessage . newBuilder ( desc ) ; List < Descriptors . FieldDescriptor > fields = new ArrayList <> ( ) ; fields . addAll ( desc . getFields ( ) ) ; if ( messageTypeToExtensionMap . containsKey ( desc . getFullName ( ) ) ) { fields . addAll ( messageTypeToExtensionMap . get ( desc . getFullName ( ) ) ) ; } // root field is always a Map in a record representing protobuf data Map < String , Field > valueAsMap = field . getValueAsMap ( ) ; for ( Descriptors . FieldDescriptor f : fields ) { Field mapField = valueAsMap . get ( f . getName ( ) ) ; // Repeated field if ( f . isMapField ( ) ) { handleMapField ( record , mapField , fieldPath , messageTypeToExtensionMap , defaultValueMap , f , builder ) ; } else if ( f . isRepeated ( ) ) { if ( mapField != null ) { handleRepeatedField ( record , mapField , fieldPath , messageTypeToExtensionMap , defaultValueMap , f , builder ) ; } } else { // non repeated field handleNonRepeatedField ( record , valueAsMap , fieldPath , messageTypeToExtensionMap , defaultValueMap , desc , f , builder ) ; } } // if record has unknown fields for this field path, handle it try { handleUnknownFields ( record , fieldPath , builder ) ; } catch ( IOException e ) { throw new DataGeneratorException ( Errors . PROTOBUF_05 , e . toString ( ) , e ) ; } return builder . build ( ) ; }


Input: Helper method to upgrade both HTTP stages to the JerseyConfigBean [CODESPLIT] public static void upgradeToJerseyConfigBean ( List < Config > configs ) { List < Config > configsToAdd = new ArrayList <> ( ) ; List < Config > configsToRemove = new ArrayList <> ( ) ; List < String > movedConfigs = ImmutableList . of ( "conf.requestTimeoutMillis" , "conf.numThreads" , "conf.authType" , "conf.oauth" , "conf.basicAuth" , "conf.useProxy" , "conf.proxy" , "conf.sslConfig" ) ; for ( Config config : configs ) { if ( hasPrefixIn ( movedConfigs , config . getName ( ) ) ) { configsToRemove . add ( config ) ; configsToAdd . add ( new Config ( config . getName ( ) . replace ( "conf." , "conf.client." ) , config . getValue ( ) ) ) ; } } configsToAdd . add ( new Config ( "conf.client.transferEncoding" , RequestEntityProcessing . CHUNKED ) ) ; configs . removeAll ( configsToRemove ) ; configs . addAll ( configsToAdd ) ; }


Input: Check network connection to the kudu master . [CODESPLIT] public static void checkConnection ( AsyncKuduClient kuduClient , Context context , String KUDU_MASTER , final List < Stage . ConfigIssue > issues ) { try { kuduClient . getTablesList ( ) . join ( ) ; } catch ( Exception ex ) { issues . add ( context . createConfigIssue ( Groups . KUDU . name ( ) , KuduLookupConfig . CONF_PREFIX + KUDU_MASTER , Errors . KUDU_00 , ex . toString ( ) , ex ) ) ; } }


Input: Convert from Kudu type to SDC Field type [CODESPLIT] public static Field . Type convertFromKuduType ( Type kuduType ) { switch ( kuduType ) { case BINARY : return Field . Type . BYTE_ARRAY ; case BOOL : return Field . Type . BOOLEAN ; case DOUBLE : return Field . Type . DOUBLE ; case FLOAT : return Field . Type . FLOAT ; case INT8 : return Field . Type . BYTE ; case INT16 : return Field . Type . SHORT ; case INT32 : return Field . Type . INTEGER ; case INT64 : return Field . Type . LONG ; case STRING : return Field . Type . STRING ; case UNIXTIME_MICROS : return Field . Type . DATETIME ; default : if ( "DECIMAL" . equals ( kuduType . name ( ) ) ) { return Field . Type . DECIMAL ; } throw new UnsupportedOperationException ( "Unknown data type: " + kuduType . getName ( ) ) ; } }


Input: Create a field and assign a value off of RowResult . [CODESPLIT] public static Field createField ( RowResult result , String fieldName , Type type ) throws StageException { switch ( type ) { case INT8 : return Field . create ( Field . Type . BYTE , result . getByte ( fieldName ) ) ; case INT16 : return Field . create ( Field . Type . SHORT , result . getShort ( fieldName ) ) ; case INT32 : return Field . create ( Field . Type . INTEGER , result . getInt ( fieldName ) ) ; case INT64 : return Field . create ( Field . Type . LONG , result . getLong ( fieldName ) ) ; case BINARY : try { return Field . create ( Field . Type . BYTE_ARRAY , result . getBinary ( fieldName ) ) ; } catch ( IllegalArgumentException ex ) { throw new OnRecordErrorException ( Errors . KUDU_35 , fieldName ) ; } case STRING : return Field . create ( Field . Type . STRING , result . getString ( fieldName ) ) ; case BOOL : return Field . create ( Field . Type . BOOLEAN , result . getBoolean ( fieldName ) ) ; case FLOAT : return Field . create ( Field . Type . FLOAT , result . getFloat ( fieldName ) ) ; case DOUBLE : return Field . create ( Field . Type . DOUBLE , result . getDouble ( fieldName ) ) ; case UNIXTIME_MICROS : //UNIXTIME_MICROS is in microsecond return Field . create ( Field . Type . DATETIME , new Date ( result . getLong ( fieldName ) / 1000L ) ) ; default : if ( "DECIMAL" . equals ( type . name ( ) ) ) { return Field . create ( Field . Type . DECIMAL , result . getDecimal ( fieldName ) ) ; } throw new StageException ( Errors . KUDU_10 , fieldName , type . getName ( ) ) ; } }


Input: Intercept given records with all the interceptors . [CODESPLIT] private List < Record > intercept ( List < Record > records , List < ? extends Interceptor > interceptors ) throws StageException { for ( Interceptor interceptor : interceptors ) { records = interceptor . intercept ( records ) ; } return records ; }


Input: Returns the text of the line . [CODESPLIT] public String getText ( ) { if ( line == null ) { line = new String ( buffer , offsetInChunk , length , charset ) ; } return line ; }


Input: Configuration [CODESPLIT] private boolean isConfigurationActive ( ConfigDef configDef , Map < String , Object > configuration ) { String dependsOn = configDef . dependsOn ( ) ; if ( ! dependsOn . isEmpty ( ) ) { Object dependsOnValue = configuration . get ( dependsOn ) ; if ( dependsOnValue != null ) { String valueStr = dependsOnValue . toString ( ) ; for ( String trigger : configDef . triggeredByValue ( ) ) { if ( valueStr . equals ( trigger ) ) { return true ; } } return false ; } return false ; } return true ; }


Input: Resolve stage aliases ( e . g . when a stage is renamed ) . [CODESPLIT] public static void resolveStageAlias ( StageLibraryTask stageLibrary , StageConfiguration stageConf ) { String aliasKey = Joiner . on ( "," ) . join ( stageConf . getLibrary ( ) , stageConf . getStageName ( ) ) ; String aliasValue = Strings . nullToEmpty ( stageLibrary . getStageNameAliases ( ) . get ( aliasKey ) ) ; if ( LOG . isTraceEnabled ( ) ) { for ( String key : stageLibrary . getStageNameAliases ( ) . keySet ( ) ) { LOG . trace ( "Stage Lib Alias: {} => {}" , key , stageLibrary . getStageNameAliases ( ) . get ( key ) ) ; } LOG . trace ( "Looking for '{}' and found '{}'" , aliasKey , aliasValue ) ; } if ( ! aliasValue . isEmpty ( ) ) { List < String > alias = Splitter . on ( "," ) . splitToList ( aliasValue ) ; if ( alias . size ( ) == 2 ) { LOG . debug ( "Converting '{}' to '{}'" , aliasKey , aliasValue ) ; stageConf . setLibrary ( alias . get ( 0 ) ) ; stageConf . setStageName ( alias . get ( 1 ) ) ; } else { LOG . error ( "Malformed stage alias: '{}'" , aliasValue ) ; } } }


Input: Resolve all stage library relevant aliases - this includes : [CODESPLIT] public static boolean resolveLibraryAliases ( StageLibraryTask stageLibrary , List < StageConfiguration > stageConfigurations ) { for ( StageConfiguration stageConf : stageConfigurations ) { String name = stageConf . getLibrary ( ) ; if ( stageLibrary . getLibraryNameAliases ( ) . containsKey ( name ) ) { stageConf . setLibrary ( stageLibrary . getLibraryNameAliases ( ) . get ( name ) ) ; } resolveStageAlias ( stageLibrary , stageConf ) ; } return true ; }


Input: Add any missing configs to the stage configuration . [CODESPLIT] public static void addMissingConfigsToStage ( StageLibraryTask stageLibrary , StageConfiguration stageConf ) { StageDefinition stageDef = stageLibrary . getStage ( stageConf . getLibrary ( ) , stageConf . getStageName ( ) , false ) ; if ( stageDef != null ) { for ( ConfigDefinition configDef : stageDef . getConfigDefinitions ( ) ) { String configName = configDef . getName ( ) ; Config config = stageConf . getConfig ( configName ) ; if ( config == null ) { Object defaultValue = configDef . getDefaultValue ( ) ; LOG . warn ( "Stage '{}' missing configuration '{}', adding with '{}' as default" , stageConf . getInstanceName ( ) , configName , defaultValue ) ; config = new Config ( configName , defaultValue ) ; stageConf . addConfig ( config ) ; } } } }


Input: Validate given stage configuration . [CODESPLIT] public static boolean validateStageConfiguration ( StageLibraryTask stageLibrary , boolean shouldBeSource , StageConfiguration stageConf , boolean notOnMainCanvas , IssueCreator issueCreator , boolean isPipelineFragment , Map < String , Object > constants , List < Issue > issues ) { boolean preview = true ; StageDefinition stageDef = stageLibrary . getStage ( stageConf . getLibrary ( ) , stageConf . getStageName ( ) , false ) ; if ( stageDef == null ) { // stage configuration refers to an undefined stage definition issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0006 , stageConf . getLibrary ( ) , stageConf . getStageName ( ) , stageConf . getStageVersion ( ) ) ) ; preview = false ; } else { if ( shouldBeSource ) { if ( stageDef . getType ( ) != StageType . SOURCE && ! isPipelineFragment ) { // first stage must be a Source issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0003 ) ) ; preview = false ; } } else { if ( ! stageLibrary . isMultipleOriginSupported ( ) && stageDef . getType ( ) == StageType . SOURCE ) { // no stage other than first stage can be a Source issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0004 ) ) ; preview = false ; } } if ( ! stageConf . isSystemGenerated ( ) && ! TextUtils . isValidName ( stageConf . getInstanceName ( ) ) ) { // stage instance name has an invalid name (it must match '[0-9A-Za-z_]+') issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0016 , stageConf . getInstanceName ( ) , TextUtils . VALID_NAME ) ) ; preview = false ; } // Hidden stages can't appear on the main canvas if ( ! notOnMainCanvas && ! stageDef . getHideStage ( ) . isEmpty ( ) ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0037 ) ) ; preview = false ; } for ( String lane : stageConf . getInputLanes ( ) ) { if ( ! TextUtils . isValidName ( lane ) ) { // stage instance input lane has an invalid name (it must match '[0-9A-Za-z_]+') issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0017 , lane , TextUtils . VALID_NAME ) ) ; preview = false ; } } for ( String lane : stageConf . getOutputLanes ( ) ) { if ( ! TextUtils . isValidName ( lane ) ) { // stage instance output lane has an invalid name (it must match '[0-9A-Za-z_]+') issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0018 , lane , TextUtils . VALID_NAME ) ) ; preview = false ; } } for ( String lane : stageConf . getEventLanes ( ) ) { if ( ! TextUtils . isValidName ( lane ) ) { // stage instance output lane has an invalid name (it must match '[0-9A-Za-z_]+') issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0100 , lane , TextUtils . VALID_NAME ) ) ; preview = false ; } } // Special validation for stage exposed limit of input lanes // -1: Means that framework is fully in control on how many input lanes should be present //  0: Means that the stage supports unlimited number of input lanes // >0: Means that stage needs exactly that amount of lanes which we will validate here if ( stageDef . getInputStreams ( ) > 0 ) { if ( stageDef . getInputStreams ( ) != stageConf . getInputLanes ( ) . size ( ) ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0094 , stageDef . getInputStreams ( ) , stageConf . getInputLanes ( ) . size ( ) ) ) ; } } // Validate proper input/output lane configuration switch ( stageDef . getType ( ) ) { case SOURCE : if ( ! stageConf . getInputLanes ( ) . isEmpty ( ) ) { // source stage cannot have input lanes issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0012 , stageDef . getType ( ) , stageConf . getInputLanes ( ) ) ) ; preview = false ; } if ( ! notOnMainCanvas && ! stageDef . isVariableOutputStreams ( ) ) { // source stage must match the output stream defined in StageDef if ( stageDef . getOutputStreams ( ) != stageConf . getOutputLanes ( ) . size ( ) ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0015 , stageDef . getOutputStreams ( ) , stageConf . getOutputLanes ( ) . size ( ) ) ) ; } } else if ( ! notOnMainCanvas && stageConf . getOutputLanes ( ) . isEmpty ( ) ) { // source stage must have at least one output lane issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0032 ) ) ; } break ; case PROCESSOR : if ( ! notOnMainCanvas && stageConf . getInputLanes ( ) . isEmpty ( ) && ! isPipelineFragment ) { // processor stage must have at least one input lane issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0014 , "Processor" ) ) ; preview = false ; } if ( ! notOnMainCanvas ) { if ( ! stageDef . isVariableOutputStreams ( ) ) { // processor stage must match the output stream defined in StageDef if ( stageDef . getOutputStreams ( ) != stageConf . getOutputLanes ( ) . size ( ) ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0015 , stageDef . getOutputStreams ( ) , stageConf . getOutputLanes ( ) . size ( ) ) ) ; } } else if ( stageConf . getOutputLanes ( ) . isEmpty ( ) ) { // processor stage must have at least one output lane issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0032 ) ) ; } } break ; case EXECUTOR : case TARGET : // Normal target stage must have at least one input lane if ( ! notOnMainCanvas && stageConf . getInputLanes ( ) . isEmpty ( ) && ! isPipelineFragment ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0014 , "Target" ) ) ; preview = false ; } // Error/Stats/Pipeline lifecycle must not have an input lane if ( notOnMainCanvas && ! stageConf . getInputLanes ( ) . isEmpty ( ) ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0012 , "Error/Stats/Lifecycle" , stageConf . getInputLanes ( ) ) ) ; preview = false ; } if ( ! stageConf . getOutputLanes ( ) . isEmpty ( ) ) { // target stage cannot have output lanes issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0013 , stageDef . getType ( ) , stageConf . getOutputLanes ( ) ) ) ; preview = false ; } if ( notOnMainCanvas && ! stageConf . getEventLanes ( ) . isEmpty ( ) ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0036 , stageDef . getType ( ) , stageConf . getEventLanes ( ) ) ) ; preview = false ; } break ; default : throw new IllegalStateException ( "Unexpected stage type " + stageDef . getType ( ) ) ; } // Validate proper event configuration if ( ! notOnMainCanvas && stageConf . getEventLanes ( ) . size ( ) > 1 ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0101 ) ) ; preview = false ; } if ( ! notOnMainCanvas && ! stageDef . isProducingEvents ( ) && stageConf . getEventLanes ( ) . size ( ) > 0 ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0102 ) ) ; preview = false ; } // Validate stage owns configuration preview &= validateComponentConfigs ( stageConf , stageDef . getConfigDefinitions ( ) , stageDef . getConfigDefinitionsMap ( ) , stageDef . getHideConfigs ( ) , stageDef . hasPreconditions ( ) , constants , issueCreator , issues ) ; // Validate service definitions Set < String > expectedServices = stageDef . getServices ( ) . stream ( ) . map ( service -> service . getService ( ) . getName ( ) ) . collect ( Collectors . toSet ( ) ) ; Set < String > configuredServices = stageConf . getServices ( ) . stream ( ) . map ( service -> service . getService ( ) . getName ( ) ) . collect ( Collectors . toSet ( ) ) ; if ( ! expectedServices . equals ( configuredServices ) ) { issues . add ( issueCreator . create ( stageConf . getInstanceName ( ) , ValidationError . VALIDATION_0200 , StringUtils . join ( expectedServices , "," ) , StringUtils . join ( configuredServices , "," ) ) ) ; preview = false ; } else { // Validate all services for ( ServiceConfiguration serviceConf : stageConf . getServices ( ) ) { ServiceDefinition serviceDef = stageLibrary . getServiceDefinition ( serviceConf . getService ( ) , false ) ; preview &= validateComponentConfigs ( serviceConf , serviceDef . getConfigDefinitions ( ) , serviceDef . getConfigDefinitionsMap ( ) , Collections . emptySet ( ) , false , constants , issueCreator . forService ( serviceConf . getService ( ) . getName ( ) ) , issues ) ; } } } return preview ; }


Input: { [CODESPLIT] @ Override protected List < ConfigIssue > init ( ) { List < ConfigIssue > issues = super . init ( ) ; errorRecordHandler = new DefaultErrorRecordHandler ( getContext ( ) ) ; // NOSONAR double rateLimit = conf . rateLimit > 0 ? ( 1000.0 / conf . rateLimit ) : Double . MAX_VALUE ; rateLimiter = RateLimiter . create ( rateLimit ) ; httpClientCommon . init ( issues , getContext ( ) ) ; conf . dataFormatConfig . init ( getContext ( ) , conf . dataFormat , Groups . HTTP . name ( ) , HttpClientCommon . DATA_FORMAT_CONFIG_PREFIX , issues ) ; bodyVars = getContext ( ) . createELVars ( ) ; bodyEval = getContext ( ) . createELEval ( REQUEST_BODY_CONFIG_NAME ) ; if ( issues . isEmpty ( ) ) { parserFactory = conf . dataFormatConfig . getParserFactory ( ) ; } return issues ; }


Input: { [CODESPLIT] @ Override public void process ( Batch batch , SingleLaneBatchMaker batchMaker ) throws StageException { List < Future < Response >> responses = new ArrayList <> ( ) ; resolvedRecords . clear ( ) ; Iterator < Record > records = batch . getRecords ( ) ; while ( records . hasNext ( ) ) { Record record = records . next ( ) ; String resolvedUrl = httpClientCommon . getResolvedUrl ( conf . resourceUrl , record ) ; WebTarget target = httpClientCommon . getClient ( ) . target ( resolvedUrl ) ; LOG . debug ( "Resolved HTTP Client URL: '{}'" , resolvedUrl ) ; // If the request (headers or body) contain a known sensitive EL and we're not using https then fail the request. if ( httpClientCommon . requestContainsSensitiveInfo ( conf . headers , conf . requestBody ) && ! target . getUri ( ) . getScheme ( ) . toLowerCase ( ) . startsWith ( "https" ) ) { throw new StageException ( Errors . HTTP_07 ) ; } // from HttpStreamConsumer final MultivaluedMap < String , Object > resolvedHeaders = httpClientCommon . resolveHeaders ( conf . headers , record ) ; String contentType = HttpStageUtil . getContentTypeWithDefault ( resolvedHeaders , conf . defaultRequestContentType ) ; final AsyncInvoker asyncInvoker = target . request ( ) . property ( OAuth1ClientSupport . OAUTH_PROPERTY_ACCESS_TOKEN , httpClientCommon . getAuthToken ( ) ) . headers ( resolvedHeaders ) . async ( ) ; HttpMethod method = httpClientCommon . getHttpMethod ( conf . httpMethod , conf . methodExpression , record ) ; rateLimiter . acquire ( ) ; if ( conf . requestBody != null && ! conf . requestBody . isEmpty ( ) && method != HttpMethod . GET ) { RecordEL . setRecordInContext ( bodyVars , record ) ; final String requestBody = bodyEval . eval ( bodyVars , conf . requestBody , String . class ) ; resolvedRecords . put ( record , new HeadersAndBody ( resolvedHeaders , requestBody , contentType , method , target ) ) ; responses . add ( asyncInvoker . method ( method . getLabel ( ) , Entity . entity ( requestBody , contentType ) ) ) ; } else { resolvedRecords . put ( record , new HeadersAndBody ( resolvedHeaders , null , null , method , target ) ) ; responses . add ( asyncInvoker . method ( method . getLabel ( ) ) ) ; } } records = batch . getRecords ( ) ; int recordNum = 0 ; while ( records . hasNext ( ) ) { try { Record record = processResponse ( records . next ( ) , responses . get ( recordNum ) , conf . maxRequestCompletionSecs , false ) ; if ( record != null ) { batchMaker . addRecord ( record ) ; } } catch ( OnRecordErrorException e ) { errorRecordHandler . onError ( e ) ; } finally { ++ recordNum ; } } if ( ! resolvedRecords . isEmpty ( ) ) { reprocessIfRequired ( batchMaker ) ; } }


Input: Waits for the Jersey client to complete an asynchronous request checks the response code and continues to parse the response if it is deemed ok . [CODESPLIT] private Record processResponse ( Record record , Future < Response > responseFuture , long maxRequestCompletionSecs , boolean failOn403 ) throws StageException { Response response = null ; try { response = responseFuture . get ( maxRequestCompletionSecs , TimeUnit . SECONDS ) ; InputStream responseBody = null ; if ( response . hasEntity ( ) ) { responseBody = response . readEntity ( InputStream . class ) ; } int responseStatus = response . getStatus ( ) ; if ( conf . client . useOAuth2 && response . getStatus ( ) == 403 && ! failOn403 ) { HttpStageUtil . getNewOAuth2Token ( conf . client . oauth2 , httpClientCommon . getClient ( ) ) ; return null ; } else if ( responseStatus < 200 || responseStatus >= 300 ) { resolvedRecords . remove ( record ) ; throw new OnRecordErrorException ( record , Errors . HTTP_01 , response . getStatus ( ) , response . getStatusInfo ( ) . getReasonPhrase ( ) + " " + responseBody ) ; } resolvedRecords . remove ( record ) ; Record parsedResponse = parseResponse ( responseBody ) ; if ( parsedResponse != null ) { record . set ( conf . outputField , parsedResponse . get ( ) ) ; addResponseHeaders ( record , response ) ; } else if ( responseBody == null && responseStatus != 204 ) { throw new OnRecordErrorException ( record , Errors . HTTP_34 ) ; } return record ; } catch ( InterruptedException | ExecutionException e ) { LOG . error ( Errors . HTTP_03 . getMessage ( ) , e . toString ( ) , e ) ; throw new OnRecordErrorException ( record , Errors . HTTP_03 , e . toString ( ) ) ; } catch ( TimeoutException e ) { LOG . error ( "HTTP request future timed out" , e . toString ( ) , e ) ; throw new OnRecordErrorException ( record , Errors . HTTP_03 , e . toString ( ) ) ; } finally { if ( response != null ) { response . close ( ) ; } } }


Input: Parses the HTTP response text from a request into SDC Records [CODESPLIT] private Record parseResponse ( InputStream response ) throws StageException { Record record = null ; if ( conf . httpMethod == HttpMethod . HEAD ) { // Head will have no body so can't be parsed.   Return an empty record. record = getContext ( ) . createRecord ( "" ) ; record . set ( Field . create ( new HashMap ( ) ) ) ; } else if ( response != null ) { try ( DataParser parser = parserFactory . getParser ( "" , response , "0" ) ) { // A response may only contain a single record, so we only parse it once. record = parser . parse ( ) ; if ( conf . dataFormat == DataFormat . TEXT ) { // Output is placed in a field "/text" so we remove it here. record . set ( record . get ( "/text" ) ) ; } } catch ( IOException | DataParserException e ) { errorRecordHandler . onError ( Errors . HTTP_00 , e . toString ( ) , e ) ; } } return record ; }


Input: Populates HTTP response headers to the configured location [CODESPLIT] private void addResponseHeaders ( Record record , Response response ) throws StageException { if ( conf . headerOutputLocation == HeaderOutputLocation . NONE ) { return ; } Record . Header header = record . getHeader ( ) ; if ( conf . headerOutputLocation == HeaderOutputLocation . FIELD ) { writeResponseHeaderToField ( record , response ) ; } else if ( conf . headerOutputLocation == HeaderOutputLocation . HEADER ) { writeResponseHeaderToRecordHeader ( response , header ) ; } }


Input: Writes HTTP response headers to the SDC Record at the configured field path . [CODESPLIT] private void writeResponseHeaderToField ( Record record , Response response ) throws StageException { if ( record . has ( conf . headerOutputField ) ) { throw new StageException ( Errors . HTTP_11 , conf . headerOutputField ) ; } Map < String , Field > headers = new HashMap <> ( response . getStringHeaders ( ) . size ( ) ) ; for ( Map . Entry < String , List < String > > entry : response . getStringHeaders ( ) . entrySet ( ) ) { if ( ! entry . getValue ( ) . isEmpty ( ) ) { String firstValue = entry . getValue ( ) . get ( 0 ) ; headers . put ( entry . getKey ( ) , Field . create ( firstValue ) ) ; } } record . set ( conf . headerOutputField , Field . create ( headers ) ) ; }


Input: Writes HTTP response headers to the SDC Record header with the configured optional prefix . [CODESPLIT] private void writeResponseHeaderToRecordHeader ( Response response , Record . Header header ) { for ( Map . Entry < String , List < String > > entry : response . getStringHeaders ( ) . entrySet ( ) ) { if ( ! entry . getValue ( ) . isEmpty ( ) ) { String firstValue = entry . getValue ( ) . get ( 0 ) ; header . setAttribute ( conf . headerAttributePrefix + entry . getKey ( ) , firstValue ) ; } } }


Input: Commit metadata content to a file to disk . [CODESPLIT] synchronized private void saveMetadata ( ) throws StageException { // 0) Validate pre-conditions if ( Files . exists ( newMetadataFile ) ) { throw new StageException ( BlobStoreError . BLOB_STORE_0010 ) ; } // 1) New content is written into a new temporary file. try ( OutputStream os = Files . newOutputStream ( newMetadataFile , StandardOpenOption . CREATE , StandardOpenOption . TRUNCATE_EXISTING ) ) { jsonMapper . writeValue ( os , metadata ) ; } catch ( IOException e ) { throw new StageException ( BlobStoreError . BLOB_STORE_0001 , e . toString ( ) , e ) ; } // 2) Old metadata is dropped try { if ( Files . exists ( metadataFile ) ) { Files . delete ( metadataFile ) ; } } catch ( IOException e ) { throw new StageException ( BlobStoreError . BLOB_STORE_0011 , e . toString ( ) , e ) ; } // 3) Rename from new to old is done try { Files . move ( newMetadataFile , metadataFile ) ; } catch ( IOException e ) { throw new StageException ( BlobStoreError . BLOB_STORE_0012 , e . toString ( ) , e ) ; } }


Input: { [CODESPLIT] @ Override protected List < ConfigIssue > init ( ) { List < ConfigIssue > issues = super . init ( ) ; errorRecordHandler = new DefaultErrorRecordHandler ( getContext ( ) ) ; // NOSONAR conf . basic . init ( getContext ( ) , Groups . HTTP . name ( ) , BASIC_CONFIG_PREFIX , issues ) ; conf . dataFormatConfig . init ( getContext ( ) , conf . dataFormat , Groups . HTTP . name ( ) , DATA_FORMAT_CONFIG_PREFIX , issues ) ; conf . init ( getContext ( ) , Groups . HTTP . name ( ) , "conf." , issues ) ; if ( conf . client . tlsConfig . isEnabled ( ) ) { conf . client . tlsConfig . init ( getContext ( ) , Groups . TLS . name ( ) , TLS_CONFIG_PREFIX , issues ) ; } resourceVars = getContext ( ) . createELVars ( ) ; resourceEval = getContext ( ) . createELEval ( RESOURCE_CONFIG_NAME ) ; bodyVars = getContext ( ) . createELVars ( ) ; bodyEval = getContext ( ) . createELEval ( REQUEST_BODY_CONFIG_NAME ) ; Calendar calendar = Calendar . getInstance ( TimeZone . getTimeZone ( ZoneId . of ( conf . timeZoneID ) ) ) ; TimeEL . setCalendarInContext ( bodyVars , calendar ) ; headerVars = getContext ( ) . createELVars ( ) ; headerEval = getContext ( ) . createELEval ( HEADER_CONFIG_NAME ) ; stopVars = getContext ( ) . createELVars ( ) ; stopEval = getContext ( ) . createELEval ( STOP_CONFIG_NAME ) ; next = null ; haveMorePages = false ; if ( conf . responseStatusActionConfigs != null ) { final String cfgName = "conf.responseStatusActionConfigs" ; final EnumSet < ResponseAction > backoffRetries = EnumSet . of ( ResponseAction . RETRY_EXPONENTIAL_BACKOFF , ResponseAction . RETRY_LINEAR_BACKOFF ) ; for ( HttpResponseActionConfigBean actionConfig : conf . responseStatusActionConfigs ) { final HttpResponseActionConfigBean prevAction = statusToActionConfigs . put ( actionConfig . getStatusCode ( ) , actionConfig ) ; if ( prevAction != null ) { issues . add ( getContext ( ) . createConfigIssue ( Groups . HTTP . name ( ) , cfgName , Errors . HTTP_17 , actionConfig . getStatusCode ( ) ) ) ; } if ( backoffRetries . contains ( actionConfig . getAction ( ) ) && actionConfig . getBackoffInterval ( ) <= 0 ) { issues . add ( getContext ( ) . createConfigIssue ( Groups . HTTP . name ( ) , cfgName , Errors . HTTP_15 ) ) ; } if ( actionConfig . getStatusCode ( ) >= 200 && actionConfig . getStatusCode ( ) < 300 ) { issues . add ( getContext ( ) . createConfigIssue ( Groups . HTTP . name ( ) , cfgName , Errors . HTTP_16 ) ) ; } } } this . timeoutActionConfig = conf . responseTimeoutActionConfig ; // Validation succeeded so configure the client. if ( issues . isEmpty ( ) ) { try { configureClient ( issues ) ; } catch ( StageException e ) { // should not happen on initial connect ExceptionUtils . throwUndeclared ( e ) ; } } return issues ; }


Input: Helper method to apply Jersey client configuration properties . [CODESPLIT] private void configureClient ( List < ConfigIssue > issues ) throws StageException { clientCommon . init ( issues , getContext ( ) ) ; if ( issues . isEmpty ( ) ) { client = clientCommon . getClient ( ) ; parserFactory = conf . dataFormatConfig . getParserFactory ( ) ; } }


Input: { [CODESPLIT] @ Override public String produce ( String lastSourceOffset , int maxBatchSize , BatchMaker batchMaker ) throws StageException { long start = System . currentTimeMillis ( ) ; int chunksToFetch = Math . min ( conf . basic . maxBatchSize , maxBatchSize ) ; Optional < String > newSourceOffset = Optional . empty ( ) ; recordCount = 0 ; setPageOffset ( lastSourceOffset ) ; setResolvedUrl ( resolveInitialUrl ( lastSourceOffset ) ) ; WebTarget target = client . target ( getResolvedUrl ( ) ) ; // If the request (headers or body) contain a known sensitive EL and we're not using https then fail the request. if ( requestContainsSensitiveInfo ( ) && ! target . getUri ( ) . getScheme ( ) . toLowerCase ( ) . startsWith ( "https" ) ) { LOG . error ( Errors . HTTP_07 . getMessage ( ) ) ; throw new StageException ( Errors . HTTP_07 ) ; } boolean uninterrupted = true ; while ( ! waitTimeExpired ( start ) && uninterrupted && ( recordCount < chunksToFetch ) ) { if ( parser != null ) { // We already have an response that we haven't finished reading. newSourceOffset = Optional . of ( parseResponse ( start , chunksToFetch , batchMaker ) ) ; } else if ( shouldMakeRequest ( ) ) { if ( conf . pagination . mode != PaginationMode . NONE ) { target = client . target ( resolveNextPageUrl ( newSourceOffset . orElse ( null ) ) ) ; // Pause between paging requests so we don't get rate limited. uninterrupted = ThreadUtil . sleep ( conf . pagination . rateLimit ) ; } makeRequest ( target ) ; if ( lastRequestTimedOut ) { String actionName = conf . responseTimeoutActionConfig . getAction ( ) . name ( ) ; LOG . warn ( "HTTPClient timed out after waiting {} ms for response from server;" + " reconnecting client and proceeding as per configured {} action" , conf . client . readTimeoutMillis , actionName ) ; reconnectClient ( ) ; return nonTerminating ( lastSourceOffset ) ; } else { newSourceOffset = processResponse ( start , chunksToFetch , batchMaker ) ; } } else if ( conf . httpMode == HttpClientMode . BATCH ) { // We are done. return null ; } else { // In polling mode, waiting for the next polling interval. uninterrupted = ThreadUtil . sleep ( SLEEP_TIME_WAITING_FOR_BATCH_SIZE_MS ) ; } } return newSourceOffset . orElse ( lastSourceOffset ) ; }


Input: Returns the URL of the next page to fetch when paging is enabled . Otherwise returns the previously configured URL . [CODESPLIT] @ VisibleForTesting String resolveNextPageUrl ( String sourceOffset ) throws ELEvalException { String url ; if ( LINK_PAGINATION . contains ( conf . pagination . mode ) && next != null ) { url = next . getUri ( ) . toString ( ) ; setResolvedUrl ( url ) ; } else if ( conf . pagination . mode == PaginationMode . BY_OFFSET || conf . pagination . mode == PaginationMode . BY_PAGE ) { if ( sourceOffset != null ) { setPageOffset ( sourceOffset ) ; } url = resourceEval . eval ( resourceVars , conf . resourceUrl , String . class ) ; } else { url = getResolvedUrl ( ) ; } return url ; }


Input: Sets the startAt EL variable in scope for the resource and request body . If the source offset is null ( origin was reset ) then the initial value from the user provided configuration is used . [CODESPLIT] private void setPageOffset ( String sourceOffset ) { if ( conf . pagination . mode == PaginationMode . NONE ) { return ; } int startAt = conf . pagination . startAt ; if ( StringUtils . isNotEmpty ( sourceOffset ) ) { startAt = HttpSourceOffset . fromString ( sourceOffset ) . getStartAt ( ) ; } resourceVars . addVariable ( START_AT , startAt ) ; bodyVars . addVariable ( START_AT , startAt ) ; }


Input: Helper method to construct an HTTP request and fetch a response . [CODESPLIT] private void makeRequest ( WebTarget target ) throws StageException { hasher = HF . newHasher ( ) ; MultivaluedMap < String , Object > resolvedHeaders = resolveHeaders ( ) ; final Invocation . Builder invocationBuilder = target . request ( ) . property ( OAuth1ClientSupport . OAUTH_PROPERTY_ACCESS_TOKEN , authToken ) . headers ( resolvedHeaders ) ; boolean keepRequesting = ! getContext ( ) . isStopped ( ) ; boolean gotNewToken = false ; while ( keepRequesting ) { long startTime = System . currentTimeMillis ( ) ; try { if ( conf . requestBody != null && ! conf . requestBody . isEmpty ( ) && conf . httpMethod != HttpMethod . GET ) { final String requestBody = bodyEval . eval ( bodyVars , conf . requestBody , String . class ) ; final String contentType = HttpStageUtil . getContentTypeWithDefault ( resolvedHeaders , conf . defaultRequestContentType ) ; hasher . putString ( requestBody , Charset . forName ( conf . dataFormatConfig . charset ) ) ; setResponse ( invocationBuilder . method ( conf . httpMethod . getLabel ( ) , Entity . entity ( requestBody , contentType ) ) ) ; } else { setResponse ( invocationBuilder . method ( conf . httpMethod . getLabel ( ) ) ) ; } LOG . debug ( "Retrieved response in {} ms" , System . currentTimeMillis ( ) - startTime ) ; lastRequestTimedOut = false ; final int status = response . getStatus ( ) ; final boolean statusOk = status >= 200 && status < 300 ; if ( conf . client . useOAuth2 && status == 403 ) { // Token may have expired if ( gotNewToken ) { LOG . error ( HTTP_21 . getMessage ( ) ) ; throw new StageException ( HTTP_21 ) ; } gotNewToken = HttpStageUtil . getNewOAuth2Token ( conf . client . oauth2 , client ) ; } else if ( ! statusOk && this . statusToActionConfigs . containsKey ( status ) ) { final HttpResponseActionConfigBean actionConf = this . statusToActionConfigs . get ( status ) ; final boolean statusChanged = lastStatus != status || lastRequestTimedOut ; keepRequesting = applyResponseAction ( actionConf , statusChanged , input -> { final StageException stageException = new StageException ( Errors . HTTP_14 , status , response . readEntity ( String . class ) ) ; LOG . error ( stageException . getMessage ( ) ) ; return stageException ; } ) ; } else { keepRequesting = false ; retryCount = 0 ; } lastStatus = status ; } catch ( Exception e ) { LOG . debug ( "Request failed after {} ms" , System . currentTimeMillis ( ) - startTime ) ; final Throwable cause = e . getCause ( ) ; if ( cause != null && ( cause instanceof TimeoutException || cause instanceof SocketTimeoutException ) ) { LOG . warn ( "{} attempting to read response in HttpClientSource: {}" , cause . getClass ( ) . getSimpleName ( ) , e . getMessage ( ) , e ) ; // read timeout; consult configured action to decide on backoff and retry strategy if ( this . timeoutActionConfig != null ) { final HttpResponseActionConfigBean actionConf = this . timeoutActionConfig ; final boolean firstTimeout = ! lastRequestTimedOut ; applyResponseAction ( actionConf , firstTimeout , input -> { LOG . error ( Errors . HTTP_18 . getMessage ( ) ) ; return new StageException ( Errors . HTTP_18 ) ; } ) ; } lastRequestTimedOut = true ; keepRequesting = false ; } else if ( cause != null && cause instanceof InterruptedException ) { LOG . error ( String . format ( "InterruptedException attempting to make request in HttpClientSource; stopping: %s" , e . getMessage ( ) ) , e ) ; keepRequesting = false ; } else { LOG . error ( String . format ( "ProcessingException attempting to make request in HttpClientSource: %s" , e . getMessage ( ) ) , e ) ; Throwable reportEx = cause != null ? cause : e ; final StageException stageException = new StageException ( Errors . HTTP_32 , reportEx . toString ( ) , reportEx ) ; LOG . error ( stageException . getMessage ( ) ) ; throw stageException ; } } keepRequesting &= ! getContext ( ) . isStopped ( ) ; } // Calculate request parameter hash currentParameterHash = hasher . hash ( ) . toString ( ) ; }


Input: Determines whether or not we should continue making additional HTTP requests in the current produce () call or whether to return the current batch . [CODESPLIT] private boolean shouldMakeRequest ( ) { final long now = System . currentTimeMillis ( ) ; boolean shouldMakeRequest = lastRequestCompletedTime == - 1 ; shouldMakeRequest |= lastRequestTimedOut ; shouldMakeRequest |= next != null ; shouldMakeRequest |= ( haveMorePages && conf . pagination . mode != PaginationMode . LINK_HEADER ) ; shouldMakeRequest |= now > lastRequestCompletedTime + conf . pollingInterval && conf . httpMode == HttpClientMode . POLLING ; shouldMakeRequest |= now > lastRequestCompletedTime && conf . httpMode == HttpClientMode . STREAMING && conf . httpMethod != HttpMethod . HEAD ; return shouldMakeRequest ; }


Input: Parses the response of a completed request into records and adds them to the batch . If more records are available in the response than we can add to the batch the response is not closed and parsing will continue on the next batch . [CODESPLIT] @ VisibleForTesting String parseResponse ( long start , int maxRecords , BatchMaker batchMaker ) throws StageException { HttpSourceOffset sourceOffset = new HttpSourceOffset ( getResolvedUrl ( ) , currentParameterHash , System . currentTimeMillis ( ) , getCurrentPage ( ) ) ; InputStream in = null ; if ( parser == null ) { // Only get a new parser if we are done with the old one. in = getResponse ( ) . readEntity ( InputStream . class ) ; try { parser = parserFactory . getParser ( sourceOffset . toString ( ) , in , "0" ) ; } catch ( DataParserException e ) { if ( e . getErrorCode ( ) == JSON_PARSER_00 ) { LOG . warn ( "No data returned in HTTP response body." , e ) ; return sourceOffset . toString ( ) ; } LOG . warn ( "Error parsing response" , e ) ; throw e ; } } Record record = null ; int subRecordCount = 0 ; try { do { record = parser . parse ( ) ; if ( record == null ) { break ; } // LINK_FIELD pagination if ( conf . pagination . mode == PaginationMode . LINK_FIELD ) { // evaluate stopping condition RecordEL . setRecordInContext ( stopVars , record ) ; haveMorePages = ! stopEval . eval ( stopVars , conf . pagination . stopCondition , Boolean . class ) ; if ( haveMorePages ) { next = Link . fromUri ( record . get ( conf . pagination . nextPageFieldPath ) . getValueAsString ( ) ) . build ( ) ; } else { next = null ; } } if ( conf . pagination . mode != PaginationMode . NONE && record . has ( conf . pagination . resultFieldPath ) ) { subRecordCount = parsePaginatedResult ( batchMaker , sourceOffset . toString ( ) , record ) ; recordCount += subRecordCount ; } else { addResponseHeaders ( record . getHeader ( ) ) ; batchMaker . addRecord ( record ) ; ++ recordCount ; } } while ( recordCount < maxRecords && ! waitTimeExpired ( start ) ) ; } catch ( IOException e ) { LOG . error ( Errors . HTTP_00 . getMessage ( ) , e . toString ( ) , e ) ; errorRecordHandler . onError ( Errors . HTTP_00 , e . toString ( ) , e ) ; } finally { try { if ( record == null ) { cleanupResponse ( in ) ; } if ( subRecordCount != 0 ) { incrementSourceOffset ( sourceOffset , subRecordCount ) ; } } catch ( IOException e ) { LOG . warn ( Errors . HTTP_28 . getMessage ( ) , e . toString ( ) , e ) ; errorRecordHandler . onError ( Errors . HTTP_28 , e . toString ( ) , e ) ; } } return sourceOffset . toString ( ) ; }


Input: Used only for HEAD requests . Sets up a record for output based on headers only with an empty body . [CODESPLIT] String parseHeadersOnly ( BatchMaker batchMaker ) throws StageException { HttpSourceOffset sourceOffset = new HttpSourceOffset ( getResolvedUrl ( ) , currentParameterHash , System . currentTimeMillis ( ) , getCurrentPage ( ) ) ; Record record = getContext ( ) . createRecord ( sourceOffset + "::0" ) ; addResponseHeaders ( record . getHeader ( ) ) ; record . set ( Field . create ( new HashMap ( ) ) ) ; batchMaker . addRecord ( record ) ; recordCount ++ ; incrementSourceOffset ( sourceOffset , 1 ) ; lastRequestCompletedTime = System . currentTimeMillis ( ) ; return sourceOffset . toString ( ) ; }


Input: Increments the current source offset s startAt portion by the specified amount . This is the number of records parsed when paging BY_OFFSET or 1 if incrementing BY_PAGE . [CODESPLIT] private void incrementSourceOffset ( HttpSourceOffset sourceOffset , int increment ) { if ( conf . pagination . mode == PaginationMode . BY_PAGE ) { sourceOffset . incrementStartAt ( 1 ) ; } else if ( conf . pagination . mode == PaginationMode . BY_OFFSET ) { sourceOffset . incrementStartAt ( increment ) ; } }


Input: Cleanup the { [CODESPLIT] private void cleanupResponse ( InputStream in ) throws IOException { IOException ex = null ; LOG . debug ( "Cleanup after request processing complete." ) ; lastRequestCompletedTime = System . currentTimeMillis ( ) ; if ( in != null ) { try { in . close ( ) ; } catch ( IOException e ) { LOG . warn ( "Error closing input stream" , ex ) ; ex = e ; } } getResponse ( ) . close ( ) ; setResponse ( null ) ; try { parser . close ( ) ; } catch ( IOException e ) { LOG . warn ( "Error closing parser" , ex ) ; ex = e ; } parser = null ; if ( ex != null ) { throw ex ; } }


Input: Returns the most recently requested page number or page offset requested . [CODESPLIT] @ VisibleForTesting int getCurrentPage ( ) { // Body params take precedence, but usually only one or the other should be used. if ( bodyVars . hasVariable ( START_AT ) ) { return ( int ) bodyVars . getVariable ( START_AT ) ; } else if ( resourceVars . hasVariable ( START_AT ) ) { return ( int ) resourceVars . getVariable ( START_AT ) ; } return 0 ; }


Input: Parses a paginated result from the configured field . [CODESPLIT] private int parsePaginatedResult ( BatchMaker batchMaker , String sourceOffset , Record record ) throws StageException { int numSubRecords = 0 ; if ( ! record . has ( conf . pagination . resultFieldPath ) ) { final StageException stageException = new StageException ( Errors . HTTP_12 , conf . pagination . resultFieldPath ) ; LOG . error ( stageException . getMessage ( ) ) ; throw stageException ; } Field resultField = record . get ( conf . pagination . resultFieldPath ) ; if ( resultField . getType ( ) != Field . Type . LIST ) { final StageException stageException = new StageException ( Errors . HTTP_08 , resultField . getType ( ) ) ; LOG . error ( stageException . getMessage ( ) ) ; throw stageException ; } List < Field > results = resultField . getValueAsList ( ) ; int subRecordIdx = 0 ; for ( Field result : results ) { Record r = getContext ( ) . createRecord ( sourceOffset + "::" + subRecordIdx ++ ) ; if ( conf . pagination . keepAllFields ) { r . set ( record . get ( ) . clone ( ) ) ; r . set ( conf . pagination . resultFieldPath , result ) ; } else { r . set ( result ) ; } addResponseHeaders ( r . getHeader ( ) ) ; batchMaker . addRecord ( r ) ; ++ numSubRecords ; } if ( conf . pagination . mode != PaginationMode . LINK_FIELD ) { haveMorePages = numSubRecords > 0 ; } return numSubRecords ; }


Input: Adds the HTTP response headers to the record header . [CODESPLIT] private void addResponseHeaders ( Record . Header header ) { final MultivaluedMap < String , String > headers = getResponse ( ) . getStringHeaders ( ) ; if ( headers == null ) { return ; } for ( Map . Entry < String , List < String > > entry : headers . entrySet ( ) ) { if ( ! entry . getValue ( ) . isEmpty ( ) ) { String firstValue = entry . getValue ( ) . get ( 0 ) ; header . setAttribute ( entry . getKey ( ) , firstValue ) ; } } }


Input: Resolves any expressions in the Header value entries of the request . [CODESPLIT] private MultivaluedMap < String , Object > resolveHeaders ( ) throws StageException { MultivaluedMap < String , Object > requestHeaders = new MultivaluedHashMap <> ( ) ; for ( Map . Entry < String , String > entry : conf . headers . entrySet ( ) ) { List < Object > header = new ArrayList <> ( 1 ) ; Object resolvedValue = headerEval . eval ( headerVars , entry . getValue ( ) , String . class ) ; header . add ( resolvedValue ) ; requestHeaders . put ( entry . getKey ( ) , header ) ; hasher . putString ( entry . getKey ( ) , Charset . forName ( conf . dataFormatConfig . charset ) ) ; hasher . putString ( entry . getValue ( ) , Charset . forName ( conf . dataFormatConfig . charset ) ) ; } return requestHeaders ; }


Input: Verifies that the response was a successful one and has data and continues to parse the response . [CODESPLIT] private Optional < String > processResponse ( long start , int maxRecords , BatchMaker batchMaker ) throws StageException { Optional < String > newSourceOffset = Optional . empty ( ) ; if ( getResponse ( ) == null ) { return newSourceOffset ; } // Response was not in the OK range, so treat as an error int status = getResponse ( ) . getStatus ( ) ; if ( status < 200 || status >= 300 ) { lastRequestCompletedTime = System . currentTimeMillis ( ) ; String reason = getResponse ( ) . getStatusInfo ( ) . getReasonPhrase ( ) ; String respString = getResponse ( ) . readEntity ( String . class ) ; getResponse ( ) . close ( ) ; setResponse ( null ) ; final String errorMsg = reason + " : " + respString ; LOG . warn ( Errors . HTTP_01 . getMessage ( ) , status , errorMsg ) ; errorRecordHandler . onError ( Errors . HTTP_01 , status , errorMsg ) ; return newSourceOffset ; } if ( conf . pagination . mode == PaginationMode . LINK_HEADER ) { next = getResponse ( ) . getLink ( "next" ) ; if ( next == null ) { haveMorePages = false ; } } if ( getResponse ( ) . hasEntity ( ) ) { newSourceOffset = Optional . of ( parseResponse ( start , maxRecords , batchMaker ) ) ; } else if ( conf . httpMethod . getLabel ( ) == "HEAD" ) { // Handle HEAD only requests, which have no body, by creating a blank record for output with headers. newSourceOffset = Optional . of ( parseHeadersOnly ( batchMaker ) ) ; } return newSourceOffset ; }


Input: Return true if and only if given property is defined with non empty non default value [CODESPLIT] protected boolean propertyDefined ( Configuration conf , String propertyName ) { String prop = conf . get ( propertyName ) ; // String property will have default empty, integer -1, we'll skip both of them return prop != null && ! prop . isEmpty ( ) && ! prop . equals ( "-1" ) ; }


Input: Change package name for DataParserException . [CODESPLIT] public static DataParserException convert ( com . streamsets . pipeline . lib . parser . DataParserException original ) { if ( original instanceof com . streamsets . pipeline . lib . parser . RecoverableDataParserException ) { return new RecoverableDataParserException ( ( ( com . streamsets . pipeline . lib . parser . RecoverableDataParserException ) original ) . getUnparsedRecord ( ) , original . getErrorCode ( ) , original . getParams ( ) ) ; } return new DataParserException ( original . getErrorCode ( ) , original . getParams ( ) ) ; }


Input: Change package name for DataGeneratorException . [CODESPLIT] public static DataGeneratorException convert ( com . streamsets . pipeline . lib . generator . DataGeneratorException original ) { return new DataGeneratorException ( original . getErrorCode ( ) , original . getParams ( ) ) ; }


Input: Tries to create a { @link CredentialsProvider } for the appropriate type of credentials supplied . [CODESPLIT] public Optional < CredentialsProvider > getCredentialsProvider ( Stage . Context context , List < Stage . ConfigIssue > issues ) { CredentialsProvider provider = null ; if ( credentialsProvider . equals ( CredentialsProviderType . DEFAULT_PROVIDER ) ) { return Optional . of ( SubscriptionAdminSettings . defaultCredentialsProviderBuilder ( ) . build ( ) ) ; } else if ( credentialsProvider . equals ( CredentialsProviderType . JSON_PROVIDER ) ) { Credentials credentials = getCredentials ( context , issues ) ; provider = new FixedCredentialsProvider ( ) { @ Nullable @ Override public Credentials getCredentials ( ) { return credentials ; } } ; } return Optional . ofNullable ( provider ) ; }


Input: Reads a JSON credentials file for a service account from and returns any errors . [CODESPLIT] private Credentials getCredentials ( Stage . Context context , List < Stage . ConfigIssue > issues ) { Credentials credentials = null ; File credentialsFile ; if ( Paths . get ( path ) . isAbsolute ( ) ) { credentialsFile = new File ( path ) ; } else { credentialsFile = new File ( context . getResourcesDirectory ( ) , path ) ; } if ( ! credentialsFile . exists ( ) || ! credentialsFile . isFile ( ) ) { LOG . error ( GOOGLE_01 . getMessage ( ) , credentialsFile . getPath ( ) ) ; issues . add ( context . createConfigIssue ( Groups . CREDENTIALS . name ( ) , CONF_CREDENTIALS_CREDENTIALS_PROVIDER , GOOGLE_01 , credentialsFile . getPath ( ) ) ) ; return null ; } try ( InputStream in = new FileInputStream ( credentialsFile ) ) { credentials = ServiceAccountCredentials . fromStream ( in ) ; } catch ( IOException | IllegalArgumentException e ) { LOG . error ( GOOGLE_02 . getMessage ( ) , e ) ; issues . add ( context . createConfigIssue ( Groups . CREDENTIALS . name ( ) , CONF_CREDENTIALS_CREDENTIALS_PROVIDER , GOOGLE_02 ) ) ; } return credentials ; }


Input: Run Pipeline preview [CODESPLIT] public PreviewInfoJson previewWithOverride ( String pipelineId , List < StageOutputJson > stageOutputsToOverrideJson , String rev , Integer batchSize , Integer batches , Boolean skipTargets , String endStage , Long timeout ) throws ApiException { Object postBody = stageOutputsToOverrideJson ; byte [ ] postBinaryBody = null ; // verify the required parameter 'pipelineId' is set if ( pipelineId == null ) { throw new ApiException ( 400 , "Missing the required parameter 'pipelineId' when calling previewWithOverride" ) ; } // verify the required parameter 'stageOutputsToOverrideJson' is set if ( stageOutputsToOverrideJson == null ) { throw new ApiException ( 400 , "Missing the required parameter 'stageOutputsToOverrideJson' when calling previewWithOverride" ) ; } // create path and map variables String path = "/v1/pipeline/{pipelineId}/preview" . replaceAll ( "\\{format\\}" , "json" ) . replaceAll ( "\\{" + "pipelineId" + "\\}" , apiClient . escapeString ( pipelineId . toString ( ) ) ) ; // query params List < Pair > queryParams = new ArrayList < Pair > ( ) ; Map < String , String > headerParams = new HashMap < String , String > ( ) ; Map < String , Object > formParams = new HashMap < String , Object > ( ) ; queryParams . addAll ( apiClient . parameterToPairs ( "" , "rev" , rev ) ) ; queryParams . addAll ( apiClient . parameterToPairs ( "" , "batchSize" , batchSize ) ) ; queryParams . addAll ( apiClient . parameterToPairs ( "" , "batches" , batches ) ) ; queryParams . addAll ( apiClient . parameterToPairs ( "" , "skipTargets" , skipTargets ) ) ; queryParams . addAll ( apiClient . parameterToPairs ( "" , "endStage" , endStage ) ) ; queryParams . addAll ( apiClient . parameterToPairs ( "" , "timeout" , timeout ) ) ; final String [ ] accepts = { "application/json" } ; final String accept = apiClient . selectHeaderAccept ( accepts ) ; final String [ ] contentTypes = { } ; final String contentType = apiClient . selectHeaderContentType ( contentTypes ) ; String [ ] authNames = new String [ ] { "basic" } ; TypeRef returnType = new TypeRef < PreviewInfoJson > ( ) { } ; return apiClient . invokeAPI ( path , "POST" , queryParams , postBody , postBinaryBody , headerParams , formParams , accept , contentType , authNames , returnType ) ; }


Input: we use this to trim the output in case of overruns [CODESPLIT] protected int copyToBuffer ( StringBuilder s , int initialLen , int startChar , int currentChar ) { int overrun = 0 ; int currentSize = s . length ( ) - initialLen ; int readSize = currentChar - startChar ; if ( maxLine > - 1 && currentSize + readSize > maxLine ) { int adjustedReadSize = maxLine - currentSize ; if ( adjustedReadSize > 0 ) { s . append ( cb , startChar , adjustedReadSize ) ; overrun = readSize - adjustedReadSize ; } else { overrun = readSize ; } } else { s . append ( cb , startChar , readSize ) ; } return overrun ; }


Input: Finds package names associated with a class loader which which are not JVM level packages . Anything inside java home are specifically excluded as well as some OS specific install locations ( MacOS ) . [CODESPLIT] private static SortedSet < String > findApplicationPackageNames ( ClassLoader cl ) { SortedSet < String > packages = new TreeSet <> ( ) ; while ( cl != null ) { if ( cl instanceof URLClassLoader ) { for ( URL url : ( ( URLClassLoader ) cl ) . getURLs ( ) ) { String path = url . getPath ( ) ; if ( ! path . startsWith ( JAVA_HOME ) && ! path . startsWith ( MACOS_JAVA_EXTENSIONS_DIR ) && path . endsWith ( JAR_FILE_SUFFIX ) ) { try { try ( ZipInputStream zip = new ZipInputStream ( url . openStream ( ) ) ) { for ( ZipEntry entry = zip . getNextEntry ( ) ; entry != null ; entry = zip . getNextEntry ( ) ) { if ( ! entry . isDirectory ( ) && entry . getName ( ) . endsWith ( CLASS_FILE_SUFFIX ) ) { // This ZipEntry represents a class. Now, what class does it represent? String className = entry . getName ( ) . replace ( ' ' , ' ' ) ; // including ".class" className = className . substring ( 0 , className . length ( ) - CLASS_FILE_SUFFIX . length ( ) ) ; if ( className . contains ( "." ) && ! className . startsWith ( STREAMSETS_PACKAGE ) ) { // must end with a . as we don't want o.a.h matching o.a.ha packages . add ( className . substring ( 0 , className . lastIndexOf ( ' ' ) ) + "." ) ; } } } } } catch ( IOException unlikely ) { // since these are local URL we will likely only // hit this if there is a corrupt jar in the classpath // which we will ignore if ( SDCClassLoader . isDebug ( ) ) { System . err . println ( "Error opening '" + url + "' : " + unlikely ) ; unlikely . printStackTrace ( ) ; } } } } } cl = cl . getParent ( ) ; } SystemPackage systemPackage = new SystemPackage ( SDCClassLoader . SYSTEM_API_CHILDREN_CLASSES ) ; Iterator < String > iterator = packages . iterator ( ) ; while ( iterator . hasNext ( ) ) { String packageName = iterator . next ( ) ; if ( systemPackage . isSystem ( packageName ) ) { iterator . remove ( ) ; } } removeLogicalDuplicates ( packages ) ; return packages ; }


Input: Traverses sorted list of packages and removes logical duplicates . For example if the set contains akka . akka . io . and akka . util . only akka . will remain . Note that if the set contains only akka . io . and akka . util . both will remain . Otherwise all of the org . apache . would devolve to org . [CODESPLIT] static void removeLogicalDuplicates ( SortedSet < String > packages ) { Iterator < String > iterator = packages . iterator ( ) ; if ( ! iterator . hasNext ( ) ) { return ; } String last = iterator . next ( ) ; while ( iterator . hasNext ( ) ) { String current = iterator . next ( ) ; if ( current . startsWith ( last ) ) { iterator . remove ( ) ; } else { last = current ; } } }


Input: new data . [CODESPLIT] protected void emptyBatch ( ) throws StageException { setBatchTime ( ) ; try { hdfsTargetConfigBean . getUGI ( ) . doAs ( new PrivilegedExceptionAction < Void > ( ) { @ Override public Void run ( ) throws Exception { hdfsTargetConfigBean . getCurrentWriters ( ) . purge ( ) ; if ( hdfsTargetConfigBean . getLateWriters ( ) != null ) { hdfsTargetConfigBean . getLateWriters ( ) . purge ( ) ; } return null ; } } ) ; } catch ( Exception ex ) { throw throwStageException ( ex ) ; } }


Input: { [CODESPLIT] @ Override protected List < ConfigIssue > init ( ) { // Validate configuration values and open any required resources. List < ConfigIssue > issues = super . init ( ) ; errorRecordHandler = new DefaultErrorRecordHandler ( getContext ( ) ) ; elEvals . init ( getContext ( ) ) ; Processor . Context context = getContext ( ) ; issues . addAll ( hikariConfigBean . validateConfigs ( context , issues ) ) ; if ( issues . isEmpty ( ) && null == dataSource ) { try { dataSource = jdbcUtil . createDataSourceForWrite ( hikariConfigBean , null , null , false , issues , Collections . emptyList ( ) , getContext ( ) ) ; } catch ( RuntimeException | SQLException | StageException e ) { LOG . debug ( "Could not connect to data source" , e ) ; issues . add ( getContext ( ) . createConfigIssue ( Groups . JDBC . name ( ) , CONNECTION_STRING , JdbcErrors . JDBC_00 , e . toString ( ) ) ) ; } } if ( issues . isEmpty ( ) ) { try { schemaWriter = JdbcSchemaWriterFactory . create ( hikariConfigBean . getConnectionString ( ) , dataSource ) ; } catch ( JdbcStageCheckedException e ) { issues . add ( getContext ( ) . createConfigIssue ( Groups . JDBC . name ( ) , CONNECTION_STRING , e . getErrorCode ( ) , e . getParams ( ) ) ) ; } schemaReader = new JdbcSchemaReader ( dataSource , schemaWriter ) ; tableCache = CacheBuilder . newBuilder ( ) . maximumSize ( 50 ) . build ( new CacheLoader < Pair < String , String > , LinkedHashMap < String , JdbcTypeInfo > > ( ) { @ Override public LinkedHashMap < String , JdbcTypeInfo > load ( Pair < String , String > pair ) throws Exception { return schemaReader . getTableSchema ( pair . getLeft ( ) , pair . getRight ( ) ) ; } } ) ; } // If issues is not empty, the UI will inform the user of each configuration issue in the list. return issues ; }


Input: finds the first main line in the chunk from the specified index position onwards [CODESPLIT] int findNextMainLine ( LiveFileChunk chunk , int startIdx ) { List < FileLine > lines = chunk . getLines ( ) ; int found = - 1 ; for ( int i = startIdx ; found == - 1 && i < lines . size ( ) ; i ++ ) { if ( pattern . matcher ( lines . get ( i ) . getText ( ) . trim ( ) ) . matches ( ) ) { found = i ; } } return found ; }


Input: it there is an incomplete multiline from a previous chunk it starts from it . [CODESPLIT] LiveFileChunk resolveChunk ( LiveFileChunk chunk ) { List < FileLine > completeLines = new ArrayList <> ( ) ; List < FileLine > chunkLines = chunk . getLines ( ) ; if ( incompleteMultiLine . length ( ) == 0 ) { incompleteMultiLineOffset = chunk . getOffset ( ) ; incompleteMultiLineTruncated = chunk . isTruncated ( ) ; } incompleteMultiLineTruncated |= chunk . isTruncated ( ) ; int pos = 0 ; int idx = findNextMainLine ( chunk , pos ) ; // while we have main lines we keep adding/compacting into the new chunk while ( idx > - 1 ) { //any multi lines up to the next main line belong to the previous main line for ( int i = pos ; i < idx ; i ++ ) { incompleteMultiLine . append ( chunkLines . get ( i ) . getText ( ) ) ; } // if we have incomplete lines, at this point they are a complete multiline, compact and add to new chunk lines if ( incompleteMultiLine . length ( ) != 0 ) { completeLines . add ( new FileLine ( incompleteMultiLineOffset , incompleteMultiLine . toString ( ) ) ) ; incompleteMultiLineOffset += incompleteMultiLine . length ( ) ; // clear the incomplete multi lines as we just used them to create a full line incompleteMultiLine . setLength ( 0 ) ; incompleteMultiLineTruncated = false ; } // add the current main line as incomplete as we still don't if it is a complete line incompleteMultiLine . append ( chunkLines . get ( idx ) . getText ( ) ) ; // find the next main line pos = idx + 1 ; idx = findNextMainLine ( chunk , pos ) ; } // lets process the left over multi lines in the chunk after the last main line. // if any they will kept to completed with lines from the next chunk. for ( int i = pos ; i < chunkLines . size ( ) ; i ++ ) { incompleteMultiLine . append ( chunkLines . get ( i ) . getText ( ) ) ; } if ( completeLines . isEmpty ( ) ) { // didn't get a complete multi line yet, we keep storing lines but return a null chunk chunk = null ; } else { // create a new chunk with all complete multi lines chunk = new LiveFileChunk ( chunk . getTag ( ) , chunk . getFile ( ) , chunk . getCharset ( ) , completeLines , incompleteMultiLineTruncated ) ; } return chunk ; }


Input: Get { [CODESPLIT] public Connection getConnection ( ) throws SQLException { if ( threadLocalConnection . get ( ) == null ) { threadLocalConnection . set ( getNewConnection ( ) ) ; } return threadLocalConnection . get ( ) ; }


Input: Close the current thread s connection [CODESPLIT] public void closeConnection ( ) { LOGGER . debug ( "Closing connection" ) ; Connection connectionToRemove = threadLocalConnection . get ( ) ; jdbcUtil . closeQuietly ( connectionToRemove ) ; if ( connectionToRemove != null ) { synchronized ( this ) { connectionsToCloseDuringDestroy . remove ( connectionToRemove ) ; } } threadLocalConnection . set ( null ) ; }


Input: InputStream [CODESPLIT] @ Override public int read ( byte [ ] b ) throws IOException { checkState ( InputStream . class ) ; return ( ( InputStream ) stream ) . read ( b ) ; }


Input: InputStream [CODESPLIT] @ Override public int read ( byte [ ] b , int offset , int len ) throws IOException { checkState ( InputStream . class ) ; return ( ( InputStream ) stream ) . read ( b , offset , len ) ; }


Input: InputStream [CODESPLIT] @ Override public long skip ( long n ) throws IOException { checkState ( InputStream . class ) ; return ( ( InputStream ) stream ) . skip ( n ) ; }


Input: ReadableByteChannel [CODESPLIT] @ Override public int read ( ByteBuffer dst ) throws IOException { checkState ( ReadableByteChannel . class ) ; return ( ( ReadableByteChannel ) stream ) . read ( dst ) ; }


Input: Add backslash to escape the | character within quoted sections of the input string . This prevents the | from being processed as part of a regex . [CODESPLIT] private static String escapeQuotedSubstring ( String input ) { String [ ] parts = input . split ( "'" ) ; StringBuilder output = new StringBuilder ( input . length ( ) * 2 ) ; for ( int i = 0 ; i < parts . length ; i ++ ) { if ( ( i % 2 ) == 1 ) { output . append ( "'" ) . append ( parts [ i ] . replace ( "|" , "\\|" ) ) . append ( "'" ) ; } else { output . append ( parts [ i ] ) ; } } return output . toString ( ) ; }


Input: Returns a flow control setting such that a subscriber will block if it has buffered more messages than can be processed in a single batch times the number of record processors . Since the flow control settings are per subscriber we should divide by the number of subscribers to avoid buffering too much data in each subscriber . [CODESPLIT] private FlowControlSettings getFlowControlSettings ( ) { return FlowControlSettings . newBuilder ( ) . setLimitExceededBehavior ( FlowController . LimitExceededBehavior . Block ) . setMaxOutstandingElementCount ( ( long ) conf . basic . maxBatchSize * conf . maxThreads / conf . advanced . numSubscribers ) . build ( ) ; }