Input: Retrieve commits for a repository starting from + sha + == Parameters : [ user ] The user to whom the repo belongs . [ repo ] The repo to look for commits into . [ sha ] The first commit to start retrieving from . If nil then retrieval starts from what the project considers as master branch . [ return_retrieved ] Should retrieved commits be returned? If not memory is saved while processing them . [ num_commits ] Number of commit to retrieve [ fork_all ] Retrieve all commits even if a repo is a fork [CODESPLIT] def ensure_commits ( user , repo , sha : nil , return_retrieved : false , num_commits : - 1 , fork_all : false ) currepo = ensure_repo ( user , repo ) unless currepo [ :forked_from ] . nil? or fork_all r = retrieve_repo ( user , repo ) return if r . nil? parent_owner = r [ 'parent' ] [ 'owner' ] [ 'login' ] parent_repo = r [ 'parent' ] [ 'name' ] ensure_fork_commits ( user , repo , parent_owner , parent_repo ) return end num_retrieved = 0 commits = [ 'foo' ] # Dummy entry for simplifying the loop below commit_acc = [ ] until commits . empty? commits = retrieve_commits ( repo , sha , user , 1 ) # This means that we retrieved the last commit page again if commits . size == 1 and commits [ 0 ] [ 'sha' ] == sha commits = [ ] end retrieved = commits . map do | c | sha = c [ 'sha' ] save { ensure_commit ( repo , c [ 'sha' ] , user ) } end # Store retrieved commits to return, if client requested so if return_retrieved commit_acc = commit_acc << retrieved end num_retrieved += retrieved . size if num_commits > 0 and num_retrieved >= num_commits break end end commit_acc . flatten . select { | x | ! x . nil? } end


Input: Get the parents for a specific commit . The commit must be first stored in the database . [CODESPLIT] def ensure_parents ( commit ) commits = db [ :commits ] parents = db [ :commit_parents ] commit [ 'parents' ] . map do | p | save do url = p [ 'url' ] . split ( / \/ / ) this = commits . first ( :sha => commit [ 'sha' ] ) parent = commits . first ( :sha => url [ 7 ] ) if parent . nil? c = retrieve_commit ( url [ 5 ] , url [ 7 ] , url [ 4 ] ) if c . nil? warn "Could not retrieve commit_parent #{url[4]}/#{url[5]} -> #{url[7]} to #{this[:sha]}" next end parent = store_commit ( c , url [ 5 ] , url [ 4 ] ) end if parent . nil? warn "Could not find #{url[4]}/#{url[5]} -> #{url[7]}, parent to commit #{this[:sha]}" next end if parents . first ( :commit_id => this [ :id ] , :parent_id => parent [ :id ] ) . nil? parents . insert ( :commit_id => this [ :id ] , :parent_id => parent [ :id ] ) info "Added commit_parent #{parent[:sha]} to commit #{this[:sha]}" else debug "Parent #{parent[:sha]} for commit #{this[:sha]} exists" end parents . first ( :commit_id => this [ :id ] , :parent_id => parent [ :id ] ) end end . select { | x | ! x . nil? } end


Input: Make sure that a commit has been associated with the provided repo == Parameters : [ user ] The user that owns the repo this commit has been submitted to [ repo ] The repo receiving the commit [ sha ] The commit SHA [CODESPLIT] def ensure_repo_commit ( user , repo , sha ) project = ensure_repo ( user , repo ) if project . nil? warn "Repo #{user}/#{repo} does not exist" return end commitid = db [ :commits ] . first ( :sha => sha ) [ :id ] exists = db [ :project_commits ] . first ( :project_id => project [ :id ] , :commit_id => commitid ) if exists . nil? db [ :project_commits ] . insert ( :project_id => project [ :id ] , :commit_id => commitid ) info "Added commit_assoc #{sha} with #{user}/#{repo}" db [ :project_commits ] . first ( :project_id => project [ :id ] , :commit_id => commitid ) else debug "Association of commit #{sha} with repo #{user}/#{repo} exists" exists end end


Input: Add ( or update ) an entry for a commit author . This method uses information in the JSON object returned by Github to add ( or update ) a user in the metadata database with a full user entry ( both Git and Github details ) . [CODESPLIT] def commit_user ( githubuser , commituser ) users = db [ :users ] name = commituser [ 'name' ] email = commituser [ 'email' ] #if is_valid_email(commituser['email']) # Github user can be null when the commit email has not been associated # with any account in Github. login = githubuser [ 'login' ] unless githubuser . nil? # web-flow is a special user reserved for web-based commits: # https://api.github.com/users/web-flow # We do not follow the process below as this user's email # (noreply@github.com) clashes other existing users' emails. if login == 'web-flow' return ensure_user_byuname ( 'web-flow' ) end return ensure_user ( "#{name}<#{email}>" , false , false ) if login . nil? dbuser = users . first ( :login => login ) byemail = users . first ( :email => email ) if dbuser . nil? # We do not have the user in the database yet added = ensure_user ( login , false , false ) # A commit user can be found by email but not # by the user name he used to commit. This probably means that the # user has probably changed his user name. Treat the user's by-email # description as valid. if added . nil? and not byemail . nil? warn "Found user #{byemail[:login]} with same email #{email} as non existing user #{login}. Assigning user #{login} to #{byemail[:login]}" return users . first ( :login => byemail [ :login ] ) end # This means that the user's login has been associated with a # Github user by the time the commit was done (and hence Github was # able to associate the commit to an account), but afterwards the # user has deleted his account (before GHTorrent processed it). # On absense of something better to do, try to find the user by email # and return a "fake" user entry. if added . nil? warn "User account for user #{login} deleted from Github" return ensure_user ( "#{name}<#{email}>" , false , false ) end if byemail . nil? users . filter ( :login => login ) . update ( :name => name ) if added [ :name ] . nil? users . filter ( :login => login ) . update ( :email => email ) if added [ :email ] . nil? else # There is a previous entry for the user, currently identified by # email. This means that the user has updated his account and now # Github is able to associate his commits with his git credentials. # As the previous entry might have already associated records, just # delete the new one and update the existing with any extra data. users . filter ( :login => login ) . delete users . filter ( :email => email ) . update ( :login => login , :company => added [ :company ] , :location => added [ :location ] , :created_at => added [ :created_at ] ) end else users . filter ( :login => login ) . update ( :name => name ) if dbuser [ :name ] . nil? users . filter ( :login => login ) . update ( :email => email ) if dbuser [ :email ] . nil? end users . first ( :login => login ) end


Input: Ensure that a user exists or fetch its latest state from Github == Parameters : [ user ] The full email address in RFC 822 format or a login name to lookup the user by [ followers ] A boolean value indicating whether to retrieve the user s followers [ orgs ] A boolean value indicating whether to retrieve the organizations the user participates into == Returns : If the user can be retrieved it is returned as a Hash . Otherwise the result is nil [CODESPLIT] def ensure_user ( user , followers = true , orgs = true ) # Github only supports alpa-nums and dashes in its usernames. # All other sympbols are treated as emails. if not user . match ( / \w \- / ) begin name , email = user . split ( "<" ) email = email . split ( ">" ) [ 0 ] name = name . strip unless name . nil? email = email . strip unless email . nil? rescue StandardError warn "Not a valid email address: #{user}" return end unless is_valid_email ( email ) warn "Extracted email(#{email}) not valid for user #{user}" end u = ensure_user_byemail ( email , name ) else u = ensure_user_byuname ( user ) ensure_user_followers ( user ) if followers ensure_orgs ( user ) if orgs end return u end


Input: Ensure that a user exists or fetch its latest state from Github == Parameters : user :: The login name to lookup the user by [CODESPLIT] def ensure_user_byuname ( user ) users = db [ :users ] usr = users . first ( :login => user ) if usr . nil? u = retrieve_user_byusername ( user ) if u . nil? warn "User #{user} does not exist" return end email = unless u [ 'email' ] . nil? if u [ 'email' ] . strip == '' then nil else u [ 'email' ] . strip end end geo = geolocate ( location : u [ 'location' ] ) users . insert ( :login => u [ 'login' ] , :name => u [ 'name' ] , :company => u [ 'company' ] , :email => email , :fake => false , :deleted => false , :type => user_type ( u [ 'type' ] ) , :long => geo [ :long ] , :lat => geo [ :lat ] , :country_code => geo [ :country_code ] , :state => geo [ :state ] , :city => geo [ :city ] , :created_at => date ( u [ 'created_at' ] ) ) info "Added user #{user}" if user_type ( u [ 'type' ] ) == 'ORG' info "User #{user} is an organization. Retrieving members" ensure_org ( u [ 'login' ] , true ) end users . first ( :login => user ) else debug "User #{user} exists" usr end end


Input: Get all followers for a user . Since we do not know when the actual follow event took place we set the created_at field to the timestamp of the method call . [CODESPLIT] def ensure_user_followers ( followed ) curuser = ensure_user ( followed , false , false ) followers = db . from ( :followers , :users ) . where ( Sequel . qualify ( 'followers' , 'follower_id' ) => Sequel . qualify ( 'users' , 'id' ) ) . where ( Sequel . qualify ( 'followers' , 'user_id' ) => curuser [ :id ] ) . select ( :login ) . all retrieve_user_followers ( followed ) . reduce ( [ ] ) do | acc , x | if followers . find { | y | y [ :login ] == x [ 'login' ] } . nil? acc << x else acc end end . map { | x | save { ensure_user_follower ( followed , x [ 'login' ] ) } } . select { | x | ! x . nil? } end


Input: Make sure that a user follows another one [CODESPLIT] def ensure_user_follower ( followed , follower , date_added = nil ) follower_user = ensure_user ( follower , false , false ) followed_user = ensure_user ( followed , false , false ) if followed_user . nil? or follower_user . nil? warn "Could not find follower #{follower} or user #{followed}" return end followers = db [ :followers ] follower_id = follower_user [ :id ] followed_id = followed_user [ :id ] follower_exists = followers . first ( :user_id => followed_id , :follower_id => follower_id ) if follower_exists . nil? added = if date_added . nil? max ( follower_user [ :created_at ] , followed_user [ :created_at ] ) else date_added end retrieved = retrieve_user_follower ( followed , follower ) if retrieved . nil? warn "Could not retrieve follower #{follower} for #{followed}" return end followers . insert ( :user_id => followed_id , :follower_id => follower_id , :created_at => added ) info "Added follower #{follower} to #{followed}" else debug "Follower #{follower} for user #{followed} exists" end unless date_added . nil? followers . filter ( :user_id => followed_id , :follower_id => follower_id ) . update ( :created_at => date ( date_added ) ) info "Updated follower #{followed} -> #{follower}, created_at -> #{date(date_added)}" end followers . first ( :user_id => followed_id , :follower_id => follower_id ) end


Input: Try to retrieve a user by email . Search the DB first fall back to Github search API if unsuccessful . [CODESPLIT] def ensure_user_byemail ( email , name ) users = db [ :users ] usr = users . first ( :email => email ) if usr . nil? u = retrieve_user_byemail ( email , name ) if u . nil? or u [ 'login' ] . nil? warn "Could not retrieve user #{email} through search API query" login = ( 0 ... 8 ) . map { 65 . + ( rand ( 25 ) ) . chr } . join users . insert ( :email => email , :name => name , :login => login , :fake => true , :deleted => false , :created_at => Time . now ) info "Added user fake #{login} -> #{email}" users . first ( :login => login ) else in_db = users . first ( :login => u [ 'login' ] ) geo = geolocate ( location : u [ 'location' ] ) if in_db . nil? users . insert ( :login => u [ 'login' ] , :name => u [ 'name' ] , :company => u [ 'company' ] , :email => u [ 'email' ] , :long => geo [ :long ] , :lat => geo [ :lat ] , :country_code => geo [ :country_code ] , :state => geo [ :state ] , :city => geo [ :city ] , :fake => false , :deleted => false , :created_at => date ( u [ 'created_at' ] ) ) info "Added user #{u['login']} (#{email}) through search API query" else in_db . update ( :name => u [ 'name' ] , :company => u [ 'company' ] , :email => u [ 'email' ] , :long => geo [ :long ] , :lat => geo [ :lat ] , :country_code => geo [ :country_code ] , :state => geo [ :state ] , :city => geo [ :city ] , :fake => false , :deleted => false , :created_at => date ( u [ 'created_at' ] ) ) debug "User #{u['login']} with email #{email} exists" end users . first ( :login => u [ 'login' ] ) end else debug "User with email #{email} exists" usr end end


Input: Ensure that a repo exists or fetch its latest state from Github [CODESPLIT] def ensure_repo ( user , repo , recursive = false ) repos = db [ :projects ] curuser = ensure_user ( user , false , false ) if curuser . nil? warn "Could not find user #{user}" return end currepo = repos . first ( :owner_id => curuser [ :id ] , :name => repo ) unless currepo . nil? debug "Repo #{user}/#{repo} exists" return refresh_repo ( user , repo , currepo ) end r = retrieve_repo ( user , repo , true ) if r . nil? warn "Could not retrieve repo #{user}/#{repo}" return end if r [ 'owner' ] [ 'login' ] != curuser [ :login ] info "Repo changed owner from #{curuser[:login]} to #{r['owner']['login']}" curuser = ensure_user ( r [ 'owner' ] [ 'login' ] , false , false ) end repos . insert ( :url => r [ 'url' ] , :owner_id => curuser [ :id ] , :name => r [ 'name' ] , :description => unless r [ 'description' ] . nil? then r [ 'description' ] [ 0 .. 254 ] else nil end , :language => r [ 'language' ] , :created_at => date ( r [ 'created_at' ] ) , :updated_at => date ( Time . now ) , :etag => unless r [ 'etag' ] . nil? then r [ 'etag' ] end ) unless r [ 'parent' ] . nil? parent_owner = r [ 'parent' ] [ 'owner' ] [ 'login' ] parent_repo = r [ 'parent' ] [ 'name' ] parent = ensure_repo ( parent_owner , parent_repo ) if parent . nil? warn "Could not find repo #{parent_owner}/#{parent_repo}, parent of: #{user}/#{repo}" repos . filter ( :owner_id => curuser [ :id ] , :name => repo ) . update ( :forked_from => - 1 ) else repos . filter ( :owner_id => curuser [ :id ] , :name => repo ) . update ( :forked_from => parent [ :id ] ) info "Repo #{user}/#{repo} is a fork of #{parent_owner}/#{parent_repo}" unless ensure_fork_point ( user , repo ) . nil? warn "Could not find fork point for #{user}/#{repo}, fork of #{parent_owner}/#{parent_repo}" end end end if recursive and not ensure_repo_recursive ( user , repo ) warn "Could retrieve #{user}/#{repo} recursively" return nil end info "Added repo #{user}/#{repo}" return repos . first ( :owner_id => curuser [ :id ] , :name => repo ) end


Input: Get details about the languages used in the repository [CODESPLIT] def ensure_languages ( owner , repo ) currepo = ensure_repo ( owner , repo ) langs = retrieve_languages ( owner , repo ) if langs . nil? or langs . empty? warn "Could not find languages for repo #{owner}/#{repo}" return end ts = Time . now langs . keys . each do | lang | db [ :project_languages ] . insert ( :project_id => currepo [ :id ] , :language => lang . downcase , :bytes => langs [ lang ] , :created_at => ts ) info "Added project_language #{owner}/#{repo} -> #{lang} (#{langs[lang]} bytes)" end db [ :project_languages ] . where ( :project_id => currepo [ :id ] ) . where ( :created_at => ts ) . all end


Input: Fast path to project forking . Retrieve all commits page by page until we reach a commit that has been registered with the parent repository . Then copy all remaining parent commits to this repo . [CODESPLIT] def ensure_fork_commits ( owner , repo , parent_owner , parent_repo ) currepo = ensure_repo ( owner , repo ) if currepo . nil? warn "Could not find repo #{owner}/#{repo}" return end parent = ensure_repo ( parent_owner , parent_repo ) if parent . nil? warn "Could not find repo #{parent_owner}/#{parent_repo}, parent of #{owner}/#{repo}" return end strategy = case when config ( :fork_commits ) . match ( / /i ) :all when config ( :fork_commits ) . match ( / /i ) :fork_point when config ( :fork_commits ) . match ( / /i ) :none else :fork_point end fork_commit = ensure_fork_point ( owner , repo ) if fork_commit . nil? or fork_commit . empty? warn "Could not find fork commit for repo #{owner}/#{repo}. Retrieving all commits." return ensure_commits ( owner , repo , fork_all : true ) end debug "Retrieving commits for fork #{owner}/#{repo}: strategy is #{strategy}" return if strategy == :none if strategy == :fork_point # Retrieve commits up to fork point (fork_commit strategy) info "Retrieving commits for #{owner}/#{repo} until fork commit #{fork_commit[:sha]}" master_branch = retrieve_default_branch ( parent_owner , parent_repo ) return if master_branch . nil? sha = master_branch found = false while not found commits = retrieve_commits ( repo , sha , owner , 1 ) # This means that we retrieved no commits if commits . size == 0 break end # This means we retrieved the last page again if commits . size == 1 and commits [ 0 ] [ 'sha' ] == sha break end for c in commits ensure_commit ( repo , c [ 'sha' ] , owner ) sha = c [ 'sha' ] if c [ 'sha' ] == fork_commit [ :sha ] found = true break end end end end if strategy == :all shared_commit = db [ :commits ] . first ( :sha => fork_commit ) copied = 0 to_copy = db . from ( :project_commits , :commits ) . where ( Sequel . qualify ( 'project_commits' , 'commit_id' ) => Sequel . qualify ( 'commits' , 'id' ) ) . where ( Sequel . qualify ( 'project_commits' , 'project_id' ) => parent [ :id ] ) . where ( 'commits.created_at < ?' , shared_commit [ :created_at ] ) . select ( Sequel . qualify ( 'commits' , 'id' ) ) to_copy . each do | c | copied += 1 begin db [ :project_commits ] . insert ( :project_id => currepo [ :id ] , :commit_id => c [ :id ] ) debug "Copied commit #{c[:sha]} #{parent_owner}/#{parent_repo} -> #{owner}/#{repo} (#{copied} total)" rescue StandardError => e warn "Could not copy commit #{c[:sha]} #{parent_owner}/#{parent_repo} -> #{owner}/#{repo} : #{e.message}" end end info "Finished copying commits from #{parent_owner}/#{parent_repo} -> #{owner}/#{repo}: #{copied} total" end end


Input: Retrieve and return the commit at which the provided fork was forked at [CODESPLIT] def ensure_fork_point ( owner , repo ) fork = ensure_repo ( owner , repo , false ) if fork [ :forked_from ] . nil? warn "Repo #{owner}/#{repo} is not a fork" return nil end # Return commit if already specified unless fork [ :forked_commit_id ] . nil? commit = db [ :commits ] . where ( :id => fork [ :forked_commit_id ] ) . first return commit unless commit . nil? end parent = db . from ( :projects , :users ) . where ( Sequel . qualify ( 'projects' , 'owner_id' ) => Sequel . qualify ( 'users' , 'id' ) ) . where ( Sequel . qualify ( 'projects' , 'id' ) => fork [ :forked_from ] ) . select ( Sequel . qualify ( 'users' , 'login' ) , Sequel . qualify ( 'projects' , 'name' ) ) . first if parent . nil? warn "Unknown parent for repo #{owner}/#{repo}" return nil end default_branch = retrieve_default_branch ( parent [ :login ] , parent [ :name ] ) # Retrieve diff between parent and fork master branch diff = retrieve_master_branch_diff ( owner , repo , default_branch , parent [ :login ] , parent [ :name ] , default_branch ) if diff . nil? or diff . empty? # Try a bit harder by refreshing the default branch default_branch = retrieve_default_branch ( parent [ :login ] , parent [ :name ] , true ) diff = retrieve_master_branch_diff ( owner , repo , default_branch , parent [ :login ] , parent [ :name ] , default_branch ) end if diff . nil? or diff . empty? # This means that the are no common ancestors between the repos # This can apparently happen when the parent repo was renamed or force-pushed # example: https://github.com/openzipkin/zipkin/compare/master...aa1wi:master warn "No common ancestor between #{parent[:login]}/#{parent[:name]} and #{owner}/#{repo}" return nil else debug "Fork #{owner}/#{repo} is #{diff['ahead_by']} commits ahead and #{diff['behind_by']} commits behind #{parent[:login]}/#{parent[:name]}" end if diff [ 'ahead_by' ] . to_i > 0 # This means that the fork has diverged, and we need to search through the fork # commit graph for the earliest commit that is shared with the parent. GitHub's # diff contains a list of divergent commits. We are sorting those by date # and select the earliest one. We do date sort instead of graph walking as this # would be prohibetively slow if the commits for the parent did not exist. earliest_diverging = diff [ 'commits' ] . sort_by { | x | x [ 'commit' ] [ 'author' ] [ 'date' ] } . first if earliest_diverging [ 'parents' ] . nil? # this means that the repo was forked from the from the parent repo's initial commit. thus, they both share an initial commit. # example: https://api.github.com/repos/btakita/pain-point/compare/master...spent:master likely_fork_point = ensure_commit ( parent [ :name ] , earliest_diverging [ 'sha' ] , parent [ 'login' ] ) else # Make sure that all likely fork points exist for the parent project # and select the latest of them. # https://github.com/gousiosg/github-mirror/compare/master...pombredanne:master likely_fork_point = earliest_diverging [ 'parents' ] . map { | x | ensure_commit ( parent [ :name ] , x [ 'sha' ] , parent [ :login ] ) } . select { | x | ! x . nil? } . sort_by { | x | x [ :created_at ] } . last end forked_sha = likely_fork_point [ :sha ] else # This means that the fork has not diverged. forked_sha = diff [ 'merge_base_commit' ] [ 'sha' ] end forked_commit = ensure_commit ( repo , forked_sha , owner ) ; debug "Fork commit for #{owner}/#{repo} is #{forked_sha}" unless forked_commit . nil? db [ :projects ] . filter ( :id => fork [ :id ] ) . update ( :forked_commit_id => forked_commit [ :id ] ) info "Repo #{owner}/#{repo} was forked at #{parent[:login]}/#{parent[:name]}:#{forked_sha}" end db [ :commits ] . where ( :sha => forked_sha ) . first end


Input: Make sure that the organizations the user participates into exist [CODESPLIT] def ensure_orgs ( user ) retrieve_orgs ( user ) . map { | o | save { ensure_participation ( user , o [ 'login' ] ) } } . select { | x | ! x . nil? } end


Input: Make sure that a user participates to the provided organization [CODESPLIT] def ensure_participation ( user , organization , members = true ) org = ensure_org ( organization , members ) if org . nil? warn "Could not find organization #{organization}" return end usr = ensure_user ( user , false , false ) org_members = db [ :organization_members ] participates = org_members . first ( :user_id => usr [ :id ] , :org_id => org [ :id ] ) if participates . nil? org_members . insert ( :user_id => usr [ :id ] , :org_id => org [ :id ] ) info "Added participation #{organization} -> #{user}" org_members . first ( :user_id => usr [ :id ] , :org_id => org [ :id ] ) else debug "Participation #{organization} -> #{user} exists" participates end end


Input: Make sure that an organization exists [CODESPLIT] def ensure_org ( organization , members = true ) org = db [ :users ] . first ( :login => organization , :type => 'org' ) if org . nil? org = ensure_user ( organization , false , false ) # Not an organization, don't go ahead if org [ :type ] != 'ORG' warn "User #{organization} is not an organization" return nil end end if members retrieve_org_members ( organization ) . map do | x | ensure_participation ( ensure_user ( x [ 'login' ] , false , false ) [ :login ] , organization , false ) end end org end


Input: Get all comments for a commit [CODESPLIT] def ensure_commit_comments ( user , repo , sha ) commit_id = db [ :commits ] . first ( :sha => sha ) [ :id ] stored_comments = db [ :commit_comments ] . filter ( :commit_id => commit_id ) commit_comments = retrieve_commit_comments ( user , repo , sha ) not_saved = commit_comments . reduce ( [ ] ) do | acc , x | if stored_comments . find { | y | y [ :comment_id ] == x [ 'id' ] } . nil? acc << x else acc end end not_saved . map { | x | save { ensure_commit_comment ( user , repo , sha , x [ 'id' ] ) } } . select { | x | ! x . nil? } end


Input: Make sure that all watchers exist for a repository [CODESPLIT] def ensure_watchers ( owner , repo ) currepo = ensure_repo ( owner , repo ) if currepo . nil? warn "Could not find repo #{owner}/#{repo} for retrieving watchers" return end watchers = db . from ( :watchers , :users ) . where ( Sequel . qualify ( 'watchers' , 'user_id' ) => Sequel . qualify ( 'users' , 'id' ) ) . where ( Sequel . qualify ( 'watchers' , 'repo_id' ) => currepo [ :id ] ) . select ( :login ) . all retrieve_watchers ( owner , repo ) . reduce ( [ ] ) do | acc , x | if watchers . find { | y | y [ :login ] == x [ 'login' ] } . nil? acc << x else acc end end . map { | x | save { ensure_watcher ( owner , repo , x [ 'login' ] ) } } . select { | x | ! x . nil? } end


Input: Make sure that a watcher / stargazer exists for a repository [CODESPLIT] def ensure_watcher ( owner , repo , watcher , date_added = nil ) project = ensure_repo ( owner , repo ) new_watcher = ensure_user ( watcher , false , false ) if new_watcher . nil? or project . nil? warn "Could not find watcher #{watcher} or repo #{owner}/#{repo}" return end watchers = db [ :watchers ] watcher_exist = watchers . first ( :user_id => new_watcher [ :id ] , :repo_id => project [ :id ] ) retrieved = retrieve_watcher ( owner , repo , watcher ) created_at = case when ( not date_added . nil? ) date ( date_added ) when ( not retrieved . nil? and not retrieved [ 'created_at' ] . nil? ) date ( retrieved [ 'created_at' ] ) else max ( date ( project [ :created_at ] ) , date ( new_watcher [ :created_at ] ) ) end if watcher_exist . nil? if retrieved . nil? warn "Could not retrieve watcher #{watcher} of repo #{owner}/#{repo}" return end watchers . insert ( :user_id => new_watcher [ :id ] , :repo_id => project [ :id ] , :created_at => date ( created_at ) ) info "Added watcher #{owner}/#{repo} -> #{watcher}" else debug "Watcher #{owner}/#{repo} -> #{watcher} exists" end w = watchers . first ( :user_id => new_watcher [ :id ] , :repo_id => project [ :id ] ) if w [ :created_at ] < created_at watchers . filter ( :user_id => new_watcher [ :id ] , :repo_id => project [ :id ] ) . update ( :created_at => date ( created_at ) ) info "Updated watcher #{owner}/#{repo} -> #{watcher}, created_at -> #{date_added}" end w end


Input: Process all pull requests [CODESPLIT] def ensure_pull_requests ( owner , repo , refresh = false ) currepo = ensure_repo ( owner , repo ) if currepo . nil? warn "Could not find repo #{owner}/#{repo} for retrieving pull requests" return end raw_pull_reqs = if refresh retrieve_pull_requests ( owner , repo , refresh = true ) else pull_reqs = db [ :pull_requests ] . filter ( :base_repo_id => currepo [ :id ] ) . all retrieve_pull_requests ( owner , repo ) . reduce ( [ ] ) do | acc , x | if pull_reqs . find { | y | y [ :pullreq_id ] == x [ 'number' ] } . nil? acc << x else acc end end end raw_pull_reqs . map { | x | save { ensure_pull_request ( owner , repo , x [ 'number' ] ) } } . select { | x | ! x . nil? } end


Input: Adds a pull request history event [CODESPLIT] def ensure_pull_request_history ( id , ts , act , actor ) user = unless actor . nil? ensure_user ( actor , false , false ) end pull_req_history = db [ :pull_request_history ] entry = if [ 'opened' , 'merged' ] . include? act pull_req_history . first ( :pull_request_id => id , :action => act ) else pull_req_history . first ( :pull_request_id => id , :created_at => ( ts - 3 ) .. ( ts + 3 ) , :action => act ) end if entry . nil? pull_req_history . insert ( :pull_request_id => id , :created_at => ts , :action => act , :actor_id => unless user . nil? then user [ :id ] end ) info "Added pullreq_event (#{id}) -> (#{act}) by (#{actor}) timestamp #{ts}" else debug "Pull request (#{id}) event (#{act}) by (#{actor}) timestamp #{ts} exists" if entry [ :actor_id ] . nil? and not user . nil? pull_req_history . where ( :pull_request_id => id , :created_at => ( ts - 3 ) .. ( ts + 3 ) , :action => act ) . update ( :actor_id => user [ :id ] ) info "Updated pull request (#{id}) event (#{act}) timestamp #{ts}, actor -> #{user[:login]}" end end end


Input: Checks whether a pull request concerns two branches of the same repository [CODESPLIT] def pr_is_intra_branch ( req ) return false unless pr_has_head_repo ( req ) if req [ 'head' ] [ 'repo' ] [ 'owner' ] [ 'login' ] == req [ 'base' ] [ 'repo' ] [ 'owner' ] [ 'login' ] and req [ 'head' ] [ 'repo' ] [ 'full_name' ] == req [ 'base' ] [ 'repo' ] [ 'full_name' ] true else false end end


Input: Produces a log message [CODESPLIT] def pr_log_msg ( req ) head = if pr_has_head_repo ( req ) req [ 'head' ] [ 'repo' ] [ 'full_name' ] else '(head deleted)' end <<-eos . gsub ( / \s / , ' ' ) . strip #{ req [ 'number' ] } #{ head } #{ req [ 'base' ] [ 'repo' ] [ 'full_name' ] } eos end


Input: Process a pull request [CODESPLIT] def ensure_pull_request ( owner , repo , pullreq_id , comments = true , commits = true , history = true , state = nil , actor = nil , created_at = nil ) pulls_reqs = db [ :pull_requests ] project = ensure_repo ( owner , repo ) if project . nil? warn "Could not find repo #{owner}/#{repo} for retrieving pull request #{pullreq_id}" return end retrieved = retrieve_pull_request ( owner , repo , pullreq_id ) if retrieved . nil? warn "Could not retrieve pull_req #{owner}/#{repo} -> #{pullreq_id}" return end base_repo = ensure_repo ( retrieved [ 'base' ] [ 'repo' ] [ 'owner' ] [ 'login' ] , retrieved [ 'base' ] [ 'repo' ] [ 'name' ] ) base_commit = ensure_commit ( retrieved [ 'base' ] [ 'repo' ] [ 'name' ] , retrieved [ 'base' ] [ 'sha' ] , retrieved [ 'base' ] [ 'repo' ] [ 'owner' ] [ 'login' ] ) if pr_is_intra_branch ( retrieved ) head_repo = base_repo head_commit = ensure_commit ( retrieved [ 'base' ] [ 'repo' ] [ 'name' ] , retrieved [ 'head' ] [ 'sha' ] , retrieved [ 'base' ] [ 'repo' ] [ 'owner' ] [ 'login' ] ) debug pr_log_msg ( retrieved ) + ' is intra-branch' else head_repo = if pr_has_head_repo ( retrieved ) ensure_repo ( retrieved [ 'head' ] [ 'repo' ] [ 'owner' ] [ 'login' ] , retrieved [ 'head' ] [ 'repo' ] [ 'name' ] ) end head_commit = if not head_repo . nil? ensure_commit ( retrieved [ 'head' ] [ 'repo' ] [ 'name' ] , retrieved [ 'head' ] [ 'sha' ] , retrieved [ 'head' ] [ 'repo' ] [ 'owner' ] [ 'login' ] ) end end pull_req_user = ensure_user ( retrieved [ 'user' ] [ 'login' ] , false , false ) merged = if retrieved [ 'merged_at' ] . nil? then false else true end closed = if retrieved [ 'closed_at' ] . nil? then false else true end pull_req = pulls_reqs . first ( :base_repo_id => project [ :id ] , :pullreq_id => pullreq_id ) if pull_req . nil? pulls_reqs . insert ( :head_repo_id => if not head_repo . nil? then head_repo [ :id ] end , :base_repo_id => if not base_repo . nil? then base_repo [ :id ] end , :head_commit_id => if not head_commit . nil? then head_commit [ :id ] end , :base_commit_id => if not base_commit . nil? then base_commit [ :id ] end , :pullreq_id => pullreq_id , :intra_branch => pr_is_intra_branch ( retrieved ) ) info 'Added ' + pr_log_msg ( retrieved ) else debug pr_log_msg ( retrieved ) + ' exists' end pull_req = pulls_reqs . first ( :base_repo_id => project [ :id ] , :pullreq_id => pullreq_id ) # Add a fake (or not so fake) issue in the issues table to serve # as root for retrieving discussion comments for this pull request issues = db [ :issues ] issue = issues . first ( :pull_request_id => pull_req [ :id ] ) if issue . nil? issues . insert ( :repo_id => base_repo [ :id ] , :assignee_id => nil , :reporter_id => nil , :issue_id => pullreq_id , :pull_request => true , :pull_request_id => pull_req [ :id ] , :created_at => date ( retrieved [ 'created_at' ] ) ) debug 'Added accompanying_issue for ' + pr_log_msg ( retrieved ) else debug 'Accompanying issue for ' + pr_log_msg ( retrieved ) + ' exists' end if history # Actions on pull requests opener = pull_req_user [ :login ] ensure_pull_request_history ( pull_req [ :id ] , date ( retrieved [ 'created_at' ] ) , 'opened' , opener ) merger = if retrieved [ 'merged_by' ] . nil? then actor else retrieved [ 'merged_by' ] [ 'login' ] end ensure_pull_request_history ( pull_req [ :id ] , date ( retrieved [ 'merged_at' ] ) , 'merged' , merger ) if ( merged && state != 'merged' ) closer = if merged then merger else actor end ensure_pull_request_history ( pull_req [ :id ] , date ( retrieved [ 'closed_at' ] ) , 'closed' , closer ) if ( closed && state != 'closed' ) ensure_pull_request_history ( pull_req [ :id ] , date ( created_at ) , state , actor ) unless state . nil? end ensure_pull_request_commits ( owner , repo , pullreq_id , pull_req , retrieved ) if commits ensure_pullreq_comments ( owner , repo , pullreq_id , pull_req ) if comments ensure_issue_comments ( owner , repo , pullreq_id , pull_req [ :id ] ) if comments pull_req end


Input: Get all forks for a project . [CODESPLIT] def ensure_forks ( owner , repo ) currepo = ensure_repo ( owner , repo ) if currepo . nil? warn "Could not find repo #{owner}/#{repo} for retrieving forks" return end existing_forks = db . from ( :projects , :users ) . where ( Sequel . qualify ( 'users' , 'id' ) => Sequel . qualify ( 'projects' , 'owner_id' ) ) . where ( Sequel . qualify ( 'projects' , 'forked_from' ) => currepo [ :id ] ) . select ( Sequel . qualify ( 'projects' , 'name' ) , :login ) . all retrieve_forks ( owner , repo ) . reduce ( [ ] ) do | acc , x | if existing_forks . find do | y | forked_repo_owner = x [ 'url' ] . split ( / \/ / ) [ 4 ] forked_repo_name = x [ 'url' ] . split ( / \/ / ) [ 5 ] y [ :login ] == forked_repo_owner && y [ :name ] == forked_repo_name end . nil? acc << x else acc end end . map { | x | save { ensure_fork ( owner , repo , x [ 'id' ] ) } } . select { | x | ! x . nil? } end


Input: Make sure that a fork is retrieved for a project [CODESPLIT] def ensure_fork ( owner , repo , fork_id ) fork = retrieve_fork ( owner , repo , fork_id ) if fork . nil? warn "Could not retrieve fork #{owner}/#{repo} -> #{fork_id}" return end fork_name = if fork [ 'full_name' ] . nil? then fork [ 'url' ] . split ( / \/ / ) [ 4 .. 5 ] . join ( '/' ) else fork [ 'full_name' ] end fork_owner = fork_name . split ( / \/ / ) [ 0 ] fork_name = fork_name . split ( / \/ / ) [ 1 ] r = ensure_repo ( fork_owner , fork_name , true ) if r . nil? warn "Could not add #{fork_owner}/#{fork_name} as fork of #{owner}/#{repo}" else info "Added fork #{fork_owner}/#{fork_name} of #{owner}/#{repo}" end r end


Input: Make sure all issues exist for a project [CODESPLIT] def ensure_issues ( owner , repo ) currepo = ensure_repo ( owner , repo ) if currepo . nil? warn "Could not find repo #{owner}/#{repo} for retrieving issues" return end issues = db [ :issues ] . filter ( :repo_id => currepo [ :id ] ) . all raw_issues = retrieve_issues ( owner , repo ) . reduce ( [ ] ) do | acc , x | if issues . find { | y | y [ :issue_id ] == x [ 'number' ] } . nil? acc << x else acc end end raw_issues . map { | x | save { ensure_issue ( owner , repo , x [ 'number' ] ) } } . select { | x | ! x . nil? } end


Input: Make sure that the issue exists [CODESPLIT] def ensure_issue ( owner , repo , issue_id , events = true , comments = true , labels = true ) issues = db [ :issues ] repository = ensure_repo ( owner , repo ) if repository . nil? warn "Could not find repo #{owner}/#{repo} for retrieving issue #{issue_id}" return end cur_issue = issues . first ( :issue_id => issue_id , :repo_id => repository [ :id ] ) retrieved = retrieve_issue ( owner , repo , issue_id ) if retrieved . nil? warn "Could not retrieve issue #{owner}/#{repo} -> #{issue_id}" return end # Pull requests and issues share the same issue_id pull_req = unless retrieved [ 'pull_request' ] . nil? or retrieved [ 'pull_request' ] [ 'patch_url' ] . nil? debug "Issue #{owner}/#{repo}->#{issue_id} is a pull request" ensure_pull_request ( owner , repo , issue_id , false , false , false ) end if cur_issue . nil? reporter = ensure_user ( retrieved [ 'user' ] [ 'login' ] , false , false ) assignee = unless retrieved [ 'assignee' ] . nil? ensure_user ( retrieved [ 'assignee' ] [ 'login' ] , false , false ) end issues . insert ( :repo_id => repository [ :id ] , :assignee_id => unless assignee . nil? then assignee [ :id ] end , :reporter_id => reporter [ :id ] , :issue_id => issue_id , :pull_request => if pull_req . nil? then false else true end , :pull_request_id => unless pull_req . nil? then pull_req [ :id ] end , :created_at => date ( retrieved [ 'created_at' ] ) ) info "Added issue #{owner}/#{repo} -> #{issue_id}" else debug "Issue #{owner}/#{repo}->#{issue_id} exists" if cur_issue [ :pull_request ] == false and not pull_req . nil? info "Updated issue #{owner}/#{repo}->#{issue_id} as pull request" issues . filter ( :issue_id => issue_id , :repo_id => repository [ :id ] ) . update ( :pull_request => true , :pull_request_id => pull_req [ :id ] ) end end ensure_issue_events ( owner , repo , issue_id ) if events ensure_issue_comments ( owner , repo , issue_id ) if comments ensure_issue_labels ( owner , repo , issue_id ) if labels issues . first ( :issue_id => issue_id , :repo_id => repository [ :id ] ) end


Input: Retrieve and process all events for an issue [CODESPLIT] def ensure_issue_events ( owner , repo , issue_id ) currepo = ensure_repo ( owner , repo ) if currepo . nil? warn "Could not find repository #{owner}/#{repo} for retrieving events for issue #{issue_id}" return end issue = ensure_issue ( owner , repo , issue_id , false , false , false ) if issue . nil? warn "Could not find issue #{owner}/#{repo} -> #{issue_id} for retrieving events" return end retrieve_issue_events ( owner , repo , issue_id ) . reduce ( [ ] ) do | acc , x | if db [ :issue_events ] . first ( :issue_id => issue [ :id ] , :event_id => x [ 'id' ] ) . nil? acc << x else acc end end . map { | x | save { ensure_issue_event ( owner , repo , issue_id , x [ 'id' ] ) } } . select { | x | ! x . nil? } end


Input: Retrieve and process + event_id + for an + issue_id + [CODESPLIT] def ensure_issue_event ( owner , repo , issue_id , event_id ) issue = ensure_issue ( owner , repo , issue_id , false , false , false ) if issue . nil? warn "Could not find issue #{owner}/#{repo} -> #{issue_id} for retrieving event #{event_id}" return end issue_event_str = "#{owner}/#{repo} -> #{issue_id}/#{event_id}" curevent = db [ :issue_events ] . first ( :issue_id => issue [ :id ] , :event_id => event_id ) if curevent . nil? retrieved = retrieve_issue_event ( owner , repo , issue_id , event_id ) if retrieved . nil? warn "Could not retrieve issue_event #{owner}/#{repo} -> #{issue_id}/#{issue_event_str}" return elsif retrieved [ 'actor' ] . nil? warn "Could not find issue_event_actor #{owner}/#{repo} -> #{issue_id}/#{issue_event_str}" return end actor = ensure_user ( retrieved [ 'actor' ] [ 'login' ] , false , false ) action_specific = case retrieved [ 'event' ] when "referenced" then retrieved [ 'commit_id' ] when "merged" then retrieved [ 'commit_id' ] when "closed" then retrieved [ 'commit_id' ] else nil end if retrieved [ 'event' ] == 'assigned' def update_assignee ( owner , repo , issue , actor ) db [ :issues ] . first ( :id => issue [ :id ] ) . update ( :assignee_id => actor [ :id ] ) info "Updated #{owner}/#{repo} -> #{issue[:id]}, assignee -> #{actor[:id]}" end if issue [ :assignee_id ] . nil? then update_assignee ( owner , repo , issue , actor ) else existing = db [ :issue_events ] . filter ( :issue_id => issue [ :id ] , :action => 'assigned' ) . order ( Sequel . desc ( :created_at ) ) . first if existing . nil? update_assignee ( owner , repo , issue , actor ) elsif date ( existing [ :created_at ] ) < date ( retrieved [ 'created_at' ] ) update_assignee ( owner , repo , issue , actor ) end end end db [ :issue_events ] . insert ( :event_id => event_id , :issue_id => issue [ :id ] , :actor_id => unless actor . nil? then actor [ :id ] end , :action => retrieved [ 'event' ] , :action_specific => action_specific , :created_at => date ( retrieved [ 'created_at' ] ) ) info "Added issue_event #{owner}/#{repo} -> #{issue_id}/#{issue_event_str}" db [ :issue_events ] . first ( :issue_id => issue [ :id ] , :event_id => event_id ) else debug "Issue event #{owner}/#{repo} -> #{issue_id}/#{issue_event_str} exists" curevent end end


Input: Retrieve and process all comments for an issue . If pull_req_id is not nil this means that we are only retrieving comments for the pull request discussion for projects that don t have issues enabled [CODESPLIT] def ensure_issue_comments ( owner , repo , issue_id , pull_req_id = nil ) currepo = ensure_repo ( owner , repo ) if currepo . nil? warn "Could not find repository #{owner}/#{repo} for retrieving issue comments for issue #{issue_id}" return end issue = if pull_req_id . nil? ensure_issue ( owner , repo , issue_id , false , false , false ) else db [ :issues ] . first ( :pull_request_id => pull_req_id ) end if issue . nil? warn "Could not find issue #{owner}/#{repo} -> #{issue_id} for retrieving issue comments" return end retrieve_issue_comments ( owner , repo , issue_id ) . reduce ( [ ] ) do | acc , x | if db [ :issue_comments ] . first ( :issue_id => issue [ :id ] , :comment_id => x [ 'id' ] ) . nil? acc << x else acc end end . map { | x | save { ensure_issue_comment ( owner , repo , issue_id , x [ 'id' ] , pull_req_id ) } } . select { | x | ! x . nil? } end


Input: Retrieve and process + comment_id + for an + issue_id + [CODESPLIT] def ensure_issue_comment ( owner , repo , issue_id , comment_id , pull_req_id = nil ) issue = if pull_req_id . nil? ensure_issue ( owner , repo , issue_id , false , false , false ) else db [ :issues ] . first ( :pull_request_id => pull_req_id ) end if issue . nil? warn "Could not find issue #{owner}/#{repo} -> #{issue_id} for retrieving comment #{comment_id}" return end issue_comment_str = "#{owner}/#{repo} -> #{issue_id}/#{comment_id}" curcomment = db [ :issue_comments ] . first ( :issue_id => issue [ :id ] , :comment_id => comment_id ) if curcomment . nil? retrieved = retrieve_issue_comment ( owner , repo , issue_id , comment_id ) if retrieved . nil? warn "Could not retrieve issue_comment #{issue_comment_str}" return end user = ensure_user ( retrieved [ 'user' ] [ 'login' ] , false , false ) db [ :issue_comments ] . insert ( :comment_id => comment_id , :issue_id => issue [ :id ] , :user_id => unless user . nil? then user [ :id ] end , :created_at => date ( retrieved [ 'created_at' ] ) ) info "Added issue_comment #{issue_comment_str}" db [ :issue_comments ] . first ( :issue_id => issue [ :id ] , :comment_id => comment_id ) else debug "Issue comment #{issue_comment_str} exists" curcomment end end


Input: Retrieve repository issue labels [CODESPLIT] def ensure_labels ( owner , repo ) currepo = ensure_repo ( owner , repo ) if currepo . nil? warn "Could not find #{owner}/#{repo} for retrieving issue labels" return end repo_labels = db [ :repo_labels ] . filter ( :repo_id => currepo [ :id ] ) . all retrieve_repo_labels ( owner , repo ) . reduce ( [ ] ) do | acc , x | if repo_labels . find { | y | y [ :name ] == x [ 'name' ] } . nil? acc << x else acc end end . map { | x | save { ensure_repo_label ( owner , repo , x [ 'name' ] ) } } . select { | x | ! x . nil? } end


Input: Retrieve a single repository issue label by name [CODESPLIT] def ensure_repo_label ( owner , repo , name ) currepo = ensure_repo ( owner , repo ) if currepo . nil? warn "Could not find #{owner}/#{repo} for retrieving label #{name}" return end label = db [ :repo_labels ] . first ( :repo_id => currepo [ :id ] , :name => name ) if label . nil? retrieved = retrieve_repo_label ( owner , repo , name ) if retrieved . nil? warn "Could not retrieve repo_label #{owner}/#{repo} -> #{name}" return end db [ :repo_labels ] . insert ( :repo_id => currepo [ :id ] , :name => name ) info "Added repo_label #{owner}/#{repo} -> #{name}" db [ :repo_labels ] . first ( :repo_id => currepo [ :id ] , :name => name ) else label end end


Input: Ensure that all labels have been assigned to the issue [CODESPLIT] def ensure_issue_labels ( owner , repo , issue_id ) issue = ensure_issue ( owner , repo , issue_id , false , false , false ) if issue . nil? warn "Could not find issue #{owner}/#{repo} -> #{issue_id} for retrieving labels" return end issue_labels = db . from ( :issue_labels , :repo_labels ) . where ( Sequel . qualify ( 'issue_labels' , 'label_id' ) => Sequel . qualify ( 'repo_labels' , 'id' ) ) . where ( Sequel . qualify ( 'issue_labels' , 'issue_id' ) => issue [ :id ] ) . select ( Sequel . qualify ( 'repo_labels' , 'name' ) ) . all retrieve_issue_labels ( owner , repo , issue_id ) . reduce ( [ ] ) do | acc , x | if issue_labels . find { | y | y [ :name ] == x [ 'name' ] } . nil? acc << x else acc end end . map { | x | save { ensure_issue_label ( owner , repo , issue [ :issue_id ] , x [ 'name' ] ) } } . select { | x | ! x . nil? } end


Input: Ensure that a specific label has been assigned to the issue [CODESPLIT] def ensure_issue_label ( owner , repo , issue_id , name ) issue = ensure_issue ( owner , repo , issue_id , false , false , false ) if issue . nil? warn "Could not find issue #{owner}/#{repo} -> #{issue_id} to assign label #{name}" return end label = ensure_repo_label ( owner , repo , name ) if label . nil? warn "Could not find repo label #{owner}/#{repo} -> #{name}" return end issue_lbl = db [ :issue_labels ] . first ( :label_id => label [ :id ] , :issue_id => issue [ :id ] ) if issue_lbl . nil? db [ :issue_labels ] . insert ( :label_id => label [ :id ] , :issue_id => issue [ :id ] , ) info "Added issue_label #{name} to issue #{owner}/#{repo} -> #{issue_id}" db [ :issue_labels ] . first ( :label_id => label [ :id ] , :issue_id => issue [ :id ] ) else debug "Issue label #{name} to issue #{owner}/#{repo} -> #{issue_id} exists" issue_lbl end end


Input: Run a block in a DB transaction . Exceptions trigger transaction rollback and are rethrown . [CODESPLIT] def transaction ( & block ) db persister result = nil start_time = Time . now begin db . transaction ( :rollback => :reraise , :isolation => :repeatable , :retry_on => @retry_on_error , :num_retries => 3 ) do result = yield block end total = Time . now . to_ms - start_time . to_ms debug "Transaction committed (#{total} ms)" result rescue StandardError => e total = Time . now . to_ms - start_time . to_ms warn "Transaction failed (#{total} ms)" raise e ensure GC . start end end


Input: Store a commit contained in a hash . First check whether the commit exists . [CODESPLIT] def store_commit ( c , repo , user ) commits = db [ :commits ] commit = commits . first ( :sha => c [ 'sha' ] ) if commit . nil? author = commit_user ( c [ 'author' ] , c [ 'commit' ] [ 'author' ] ) commiter = commit_user ( c [ 'committer' ] , c [ 'commit' ] [ 'committer' ] ) repository = ensure_repo ( user , repo ) if repository . nil? warn "Could not find repo #{user}/#{repo} for storing commit #{c['sha']}" end commits . insert ( :sha => c [ 'sha' ] , :author_id => author [ :id ] , :committer_id => commiter [ :id ] , :project_id => if repository . nil? then nil else repository [ :id ] end , :created_at => date ( c [ 'commit' ] [ 'author' ] [ 'date' ] ) ) info "Added commit #{user}/#{repo} -> #{c['sha']} " commits . first ( :sha => c [ 'sha' ] ) else debug "Commit #{user}/#{repo} -> #{c['sha']} exists" commit end end


Input: Dates returned by Github are formatted as : - yyyy - mm - ddThh : mm : ssZ - yyyy / mm / dd hh : mm : ss { + / - } hhmm [CODESPLIT] def date ( arg ) if arg . class != Time time_non_zero ( Time . parse ( arg ) ) else time_non_zero ( arg ) end end


Input: Default logger [CODESPLIT] def loggerr @logger ||= proc do @logger_uniq ||= config ( :logging_uniq ) logger = if config ( :logging_file ) . casecmp ( 'stdout' ) Logger . new ( STDOUT ) elsif config ( :logging_file ) . casecmp ( 'stderr' ) Logger . new ( STDERR ) else Logger . new ( config ( :logging_file ) ) end logger . level = case config ( :logging_level ) . downcase when 'debug' then Logger :: DEBUG when 'info' then Logger :: INFO when 'warn' then Logger :: WARN when 'error' then Logger :: ERROR else Logger :: INFO end logger . formatter = proc do | severity , time , progname , msg | if progname . nil? or progname . empty? progname = @logger_uniq end "#{severity}, #{time.iso8601}, #{progname} -- #{msg}\n" end logger end . call @logger end


Input: Log a message at the given level . [CODESPLIT] def log ( level , msg ) case level when :fatal then loggerr . fatal ( retrieve_caller + msg ) when :error then loggerr . error ( retrieve_caller + msg ) when :warn then loggerr . warn ( retrieve_caller + msg ) when :info then loggerr . info ( retrieve_caller + msg ) when :debug then loggerr . debug ( retrieve_caller + msg ) else loggerr . debug ( retrieve_caller + msg ) end end


Input: A paged request . Used when the result can expand to more than one result pages . [CODESPLIT] def paged_api_request ( url , pages = config ( :mirror_history_pages_back ) , last = nil ) url = ensure_max_per_page ( url ) data = api_request_raw ( url ) return [ ] if data . nil? unless data . meta [ 'link' ] . nil? links = parse_links ( data . meta [ 'link' ] ) last = links [ 'last' ] if last . nil? if pages > 0 pages = pages - 1 if pages == 0 return parse_request_result ( data ) end end if links [ 'next' ] . nil? parse_request_result ( data ) else parse_request_result ( data ) | paged_api_request ( links [ 'next' ] , pages , last ) end else parse_request_result ( data ) end end


Input: Check whether the resource identified by the provided url has changed [CODESPLIT] def last_updated ( url , etag ) begin ts = Time . now response = do_request ( url , '' , etag ) info "Successful etag request. URL: #{url}, Etag: #{etag}, Remaining: #{@remaining}, Total: #{Time.now.to_ms - ts.to_ms} ms" rescue OpenURI :: HTTPError => e response = e . io if response . status . first != '304' etag_request_error_message ( url , e , etag ) raise e end end return Time . parse ( response . meta [ 'last-modified' ] ) unless response . meta [ 'last-modified' ] . nil? return Time . at ( 86400 ) end


Input: Determine the number of pages contained in a multi - page API response [CODESPLIT] def num_pages ( url ) url = ensure_max_per_page ( url ) data = api_request_raw ( url ) if data . nil? or data . meta . nil? or data . meta [ 'link' ] . nil? return 1 end links = parse_links ( data . meta [ 'link' ] ) if links . nil? or links [ 'last' ] . nil? return 1 end params = CGI :: parse ( URI :: parse ( links [ 'last' ] ) . query ) params [ 'page' ] [ 0 ] . to_i end


Input: Parse a Github link header [CODESPLIT] def parse_links ( links ) links . split ( / / ) . reduce ( { } ) do | acc , x | matches = x . strip . match ( / \" \" / ) acc [ matches [ 2 ] ] = matches [ 1 ] acc end end


Input: Parse the JSON result array [CODESPLIT] def parse_request_result ( result ) if result . nil? [ ] else json = result . read if json . nil? [ ] else r = JSON . parse ( json ) # Add the etag to the response only for individual entities if result . meta [ 'etag' ] and r . class != Array r [ 'etag' ] = result . meta [ 'etag' ] end r end end end


Input: Do the actual request and return the result object [CODESPLIT] def api_request_raw ( url , media_type = '' ) begin start_time = Time . now contents = do_request ( url , media_type ) total = Time . now . to_ms - start_time . to_ms info "Successful request. URL: #{url}, Remaining: #{@remaining}, Total: #{total} ms" contents rescue OpenURI :: HTTPError => e @remaining = e . io . meta [ 'x-ratelimit-remaining' ] . to_i @reset = e . io . meta [ 'x-ratelimit-reset' ] . to_i case e . io . status [ 0 ] . to_i # The following indicate valid Github return codes when 400 , # Bad request 403 , # Forbidden 404 , # Not found 409 , # Conflict -- returned on gets of empty repos 422 then # Unprocessable entity warn request_error_msg ( url , e ) return nil when 401 # Unauthorized warn request_error_msg ( url , e ) warn "Unauthorised request with token: #{@token}" raise e when 451 # DMCA takedown warn request_error_msg ( url , e ) warn "Repo was taken down (DMCA)" return nil else # Server error or HTTP conditions that Github does not report warn request_error_msg ( url , e ) raise e end rescue StandardError => e warn error_msg ( url , e ) raise e ensure # The exact limit is only enforced upon the first @reset # No idea how many requests are available on this key. Sleep if we have run out if @remaining < @req_limit to_sleep = @reset - Time . now . to_i + 2 warn "Request limit reached, reset in: #{to_sleep} secs" t = Thread . new do slept = 0 while true do debug "Sleeping for #{to_sleep - slept} seconds" sleep 1 slept += 1 end end sleep ( [ 0 , to_sleep ] . max ) t . exit end end end


Input: Attach to a specific IP address if the machine has multiple [CODESPLIT] def attach_to ( ip ) TCPSocket . instance_eval do ( class << self ; self ; end ) . instance_eval do alias_method :original_open , :open case RUBY_VERSION when / / , / / define_method ( :open ) do | conn_address , conn_port | original_open ( conn_address , conn_port , ip ) end else define_method ( :open ) do | conn_address , conn_port , local_host , local_port | original_open ( conn_address , conn_port , ip , local_port ) end end end end result = begin yield rescue StandardError => e raise e ensure TCPSocket . instance_eval do ( class << self ; self ; end ) . instance_eval do alias_method :open , :original_open remove_method :original_open end end end result end


Input: Factory method for retrieving persistence connections . The + settings + argument is a fully parsed YAML document passed on to adapters . The available + adapter + are mongo and noop [CODESPLIT] def connect ( adapter , settings ) driver = ADAPTERS [ adapter . intern ] driver . new ( settings ) end


Input: Try Github user search by email . This is optional info so it may not return any data . If this fails try searching by name http : // developer . github . com / v3 / search / #email - search [CODESPLIT] def retrieve_user_byemail ( email , name ) url = ghurl ( "legacy/user/email/#{CGI.escape(email)}" ) byemail = api_request ( url ) if byemail . nil? or byemail . empty? # Only search by name if name param looks like a proper name byname = if not name . nil? and name . split ( / / ) . size > 1 url = ghurl ( "legacy/user/search/#{CGI.escape(name)}" ) api_request ( url ) end if byname . nil? or byname [ 'users' ] . nil? or byname [ 'users' ] . empty? nil else user = byname [ 'users' ] . find do | u | u [ 'name' ] == name and not u [ 'login' ] . nil? and not retrieve_user_byusername ( u [ 'login' ] ) . nil? end unless user . nil? # Make extra sure that if we got an email it matches that # of the retrieved user if not email . nil? and user [ 'email' ] == email user else warn "Could not find user #{email}" nil end else warn "Could not find user #{email}" nil end end else unless byemail [ 'user' ] [ 'login' ] . nil? info "Added user #{byemail['user']['login']} retrieved by email #{email}" retrieve_user_byusername ( byemail [ 'user' ] [ 'login' ] ) else u = byemail [ 'user' ] unq = persister . store ( :users , u ) what = user_type ( u [ 'type' ] ) info "Added user #{what} #{user}" u end end end


Input: Retrieve a single commit from a repo [CODESPLIT] def retrieve_commit ( repo , sha , user ) commit = persister . find ( :commits , { 'sha' => "#{sha}" } ) if commit . empty? url = ghurl "repos/#{user}/#{repo}/commits/#{sha}" c = api_request ( url ) if c . nil? or c . empty? return end # commit patches are big and not always interesting if config ( :commit_handling ) == 'trim' c [ 'files' ] . each { | file | file . delete ( 'patch' ) } end persister . store ( :commits , c ) info "Added commit #{user}/#{repo} -> #{sha}" c else debug "Commit #{user}/#{repo} -> #{sha} exists" commit . first end end


Input: Retrieve commits starting from the provided + sha + [CODESPLIT] def retrieve_commits ( repo , sha , user , pages = - 1 ) url = if sha . nil? ghurl "repos/#{user}/#{repo}/commits" else ghurl "repos/#{user}/#{repo}/commits?sha=#{sha}" end commits = restricted_page_request ( url , pages ) commits . map do | c | retrieve_commit ( repo , c [ 'sha' ] , user ) end . select { | x | not x . nil? } end


Input: Retrieve organizations the provided user participates into [CODESPLIT] def retrieve_orgs ( user ) url = ghurl "users/#{user}/orgs" orgs = paged_api_request ( url ) orgs . map { | o | retrieve_org ( o [ 'login' ] ) } end


Input: Retrieve organization members [CODESPLIT] def retrieve_org_members ( org ) stored_org_members = persister . find ( :org_members , { 'org' => org } ) org_members = paged_api_request ( ghurl "orgs/#{org}/members" ) org_members . each do | x | x [ 'org' ] = org exists = ! stored_org_members . find { | f | f [ 'org' ] == org && f [ 'login' ] == x [ 'login' ] } . nil? if not exists persister . store ( :org_members , x ) info "Added org_member #{org} -> #{x['login']}" else debug "Org Member #{org} -> #{x['login']} exists" end end persister . find ( :org_members , { 'org' => org } ) . map { | o | retrieve_org ( o [ 'login' ] ) } end


Input: Retrieve all comments for a single commit [CODESPLIT] def retrieve_commit_comments ( owner , repo , sha ) retrieved_comments = paged_api_request ( ghurl "repos/#{owner}/#{repo}/commits/#{sha}/comments" ) retrieved_comments . each { | x | if persister . find ( :commit_comments , { 'commit_id' => x [ 'commit_id' ] , 'id' => x [ 'id' ] } ) . empty? persister . store ( :commit_comments , x ) end } persister . find ( :commit_comments , { 'commit_id' => sha } ) end


Input: Retrieve a single comment [CODESPLIT] def retrieve_commit_comment ( owner , repo , sha , id ) comment = persister . find ( :commit_comments , { 'commit_id' => sha , 'id' => id } ) . first if comment . nil? r = api_request ( ghurl "repos/#{owner}/#{repo}/comments/#{id}" ) if r . nil? or r . empty? warn "Could not find commit_comment #{id}. Deleted?" return end persister . store ( :commit_comments , r ) info "Added commit_comment #{r['commit_id']} -> #{r['id']}" persister . find ( :commit_comments , { 'commit_id' => sha , 'id' => id } ) . first else debug "Commit comment #{comment['commit_id']} -> #{comment['id']} exists" comment end end


Input: Retrieve all watchers for a repository [CODESPLIT] def retrieve_watchers ( user , repo ) repo_bound_items ( user , repo , :watchers , [ "repos/#{user}/#{repo}/stargazers" ] , { 'repo' => repo , 'owner' => user } , 'login' , item = nil , refresh = false , order = :desc ) end


Input: Retrieve a single watcher for a repository [CODESPLIT] def retrieve_watcher ( user , repo , watcher ) repo_bound_item ( user , repo , watcher , :watchers , [ "repos/#{user}/#{repo}/stargazers" ] , { 'repo' => repo , 'owner' => user } , 'login' , order = :desc ) end


Input: Get all events for the specified repo . GitHub will only return 90 days of events [CODESPLIT] def get_repo_events ( owner , repo ) url = ghurl ( "repos/#{owner}/#{repo}/events" ) r = paged_api_request ( url ) r . each do | e | unless get_event ( e [ 'id' ] ) . empty? debug "Repository event #{owner}/#{repo} -> #{e['type']}-#{e['id']} already exists" else persister . store ( :events , e ) info "Added event for repository #{owner}/#{repo} -> #{e['type']}-#{e['id']}" end end persister . find ( :events , { 'repo.name' => "#{owner}/#{repo}" } ) end


Input: Retrieve diff between two branches . If either branch name is not provided the branch name is resolved to the corresponding default branch [CODESPLIT] def retrieve_master_branch_diff ( owner , repo , branch , parent_owner , parent_repo , parent_branch ) branch = retrieve_default_branch ( owner , repo ) if branch . nil? parent_branch = retrieve_default_branch ( parent_owner , parent_repo ) if parent_branch . nil? return nil if branch . nil? or parent_branch . nil? cmp_url = "https://api.github.com/repos/#{parent_owner}/#{parent_repo}/compare/#{parent_branch}...#{owner}:#{branch}" api_request ( cmp_url ) end


Input: Retrieve the default branch for a repo . If nothing is retrieved master is returned [CODESPLIT] def retrieve_default_branch ( owner , repo , refresh = false ) retrieved = retrieve_repo ( owner , repo , refresh ) return nil if retrieved . nil? master_branch = 'master' if retrieved [ 'default_branch' ] . nil? # The currently stored repo entry has been created before the # default_branch field was added to the schema retrieved = retrieve_repo ( owner , repo , true ) return nil if retrieved . nil? end master_branch = retrieved [ 'default_branch' ] unless retrieved . nil? master_branch end


Input: Specify and parse top - level command line options . [CODESPLIT] def process_options command = self @options = Trollop :: options ( command . args ) do command . prepare_options ( self ) banner <<-END END opt :config , 'config.yaml file location' , :short => 'c' , :default => 'config.yaml' opt :verbose , 'verbose mode' , :short => 'v' opt :addr , 'IP address to use for performing requests' , :short => 'a' , :type => String opt :token , 'GitHub OAuth token' , :type => String , :short => 't' opt :req_limit , 'Number or requests to leave on any provided account (in reqs/hour)' , :type => Integer , :short => 'l' opt :uniq , 'Unique name for this command. Will appear in logs.' , :type => String , :short => 'u' end end


Input:
Examine the validity of the provided options in the context of the executed command . Subclasses can also call super to also invoke the checks provided by this class . [CODESPLIT] def validate if options [ :config ] . nil? unless ( File . exist? ( "config.yaml" ) ) Trollop :: die "No config file in default location (#{Dir.pwd}). You
                        need to specify the #{:config} parameter. Read the
                        documentation on how to create a config.yaml file." end else Trollop :: die "Cannot find file #{options[:config]}" unless File . exist? ( options [ :config ] ) end unless @options [ :user ] . nil? if not Process . uid == 0 Trollop :: die "Option --user (-u) can only be specified by root" end begin Etc . getpwnam ( @options [ :user ] ) rescue ArgumentError Trollop :: die "No such user: #{@options[:user]}" end end end


Input: Specify a handler to incoming messages from a connection to a queue . [CODESPLIT] def queue_client ( queue , key = queue , ack = :after , block ) stopped = false while not stopped begin conn = Bunny . new ( :host => config ( :amqp_host ) , :port => config ( :amqp_port ) , :username => config ( :amqp_username ) , :password => config ( :amqp_password ) ) conn . start ch = conn . create_channel debug "Queue setting prefetch to #{config(:amqp_prefetch)}" ch . prefetch ( config ( :amqp_prefetch ) ) debug "Queue connection to #{config(:amqp_host)} succeeded" x = ch . topic ( config ( :amqp_exchange ) , :durable => true , :auto_delete => false ) q = ch . queue ( queue , :durable => true ) q . bind ( x , :routing_key => key ) q . subscribe ( :block => true , :manual_ack => true ) do | delivery_info , properties , msg | if ack == :before ch . acknowledge ( delivery_info . delivery_tag ) end begin block . call ( msg ) ensure if ack != :before ch . acknowledge ( delivery_info . delivery_tag ) end end end rescue Bunny :: TCPConnectionFailed => e warn "Connection to #{config(:amqp_host)} failed. Retrying in 1 sec" sleep ( 1 ) rescue Bunny :: PossibleAuthenticationFailureError => e warn "Could not authenticate as #{conn.username}" rescue Bunny :: NotFound , Bunny :: AccessRefused , Bunny :: PreconditionFailed => e warn "Channel error: #{e}. Retrying in 1 sec" sleep ( 1 ) rescue Interrupt => _ stopped = true rescue StandardError => e raise e end end ch . close unless ch . nil? conn . close unless conn . nil? end


Input: Read the value for a key whose format is foo . bar . baz from a hierarchical map where a dot represents one level deep in the hierarchy . [CODESPLIT] def read_value ( from , key ) return from if key . nil? or key == "" key . split ( / \. / ) . reduce ( { } ) do | acc , x | unless acc . nil? if acc . empty? # Initial run acc = from [ x ] else if acc . has_key? ( x ) acc = acc [ x ] else # Some intermediate key does not exist return nil end end else # Some intermediate key returned a null value # This indicates a malformed entry return nil end end end


Input: Overwrite an existing + key + whose format is foo . bar ( where a dot represents one level deep in the hierarchy ) in hash + to + with + value + . If the key does not exist it will be added at the appropriate depth level [CODESPLIT] def write_value ( to , key , value ) return to if key . nil? or key == "" prev = nil key . split ( / \. / ) . reverse . each { | x | a = Hash . new a [ x ] = if prev . nil? then value else prev end prev = a a } to . merge_recursive ( prev ) end


Input: Given a location string it returns a hash like the following : [CODESPLIT] def geolocate ( location : nil , wait : config ( :geolocation_wait ) . to_i , from_cache : true ) return EMPTY_LOCATION if location . nil? or location == '' location = location_filter ( location ) geo = [ ] if from_cache geo = persister . find ( :geo_cache , { 'key' => location } ) end if geo . empty? if config ( :geolocation_service ) == 'gmaps' self . class . send :include , GHTorrent :: Geolocator :: GMaps elsif config ( :geolocation_service ) == 'bing' self . class . send :include , GHTorrent :: Geolocator :: Bing else self . class . send :include , GHTorrent :: Geolocator :: OSM end begin ts = Time . now url = format_url ( location ) req = open ( url ) p = JSON . parse ( req . read ) geo = parse_geolocation_result ( location , p ) info "Successful geolocation request. Location: #{location}" rescue StandardError => e warn "Failed geolocation request. Location: #{location}" geo = EMPTY_LOCATION geo [ :key ] = location ensure in_db_geo = persister . find ( :geo_cache , { 'key' => location } ) . first if in_db_geo . nil? begin geo [ :updated_at ] = Time . now persister . store ( :geo_cache , geo ) rescue StandardError => e warn "Could not save location #{location} -> #{geo}: #{e.message}" end end info "Added location key '#{location}' -> #{geo[:status]}" taken = Time . now . to_f - ts . to_f to_sleep = wait - taken sleep ( to_sleep ) if to_sleep > 0 end else geo = geo [ 0 ] debug "Location with key '#{location}' exists" end geo end


Input: Standard filtering on all locations used by GHTorrent [CODESPLIT] def location_filter ( location ) return nil if location . nil? location . strip . downcase . tr ( '#"<>[]' , '' ) . gsub ( / \/ / , '' ) . gsub ( / / , ' ' ) . gsub ( / / , '\1' ) end


Input: Checks if we met all the topics requirements . It will fail if we didn t send a message to a registered required topic etc . [CODESPLIT] def validate_usage! registered_topics = self . class . topics . map do | name , topic | topic . to_h . merge! ( usage_count : messages_buffer [ name ] &. count || 0 ) end used_topics = messages_buffer . map do | name , usage | topic = self . class . topics [ name ] || Responders :: Topic . new ( name , registered : false ) topic . to_h . merge! ( usage_count : usage . count ) end result = Karafka :: Schemas :: ResponderUsage . call ( registered_topics : registered_topics , used_topics : used_topics ) return if result . success? raise Karafka :: Errors :: InvalidResponderUsageError , result . errors end


Input: Checks if we met all the options requirements before sending them to the producer . [CODESPLIT] def validate_options! return true unless self . class . options_schema messages_buffer . each_value do | messages_set | messages_set . each do | message_data | result = self . class . options_schema . call ( message_data . last ) next if result . success? raise Karafka :: Errors :: InvalidResponderMessageOptionsError , result . errors end end end


Input: Takes all the messages from the buffer and delivers them one by one [CODESPLIT] def deliver! messages_buffer . each_value do | data_elements | data_elements . each do | data , options | # We map this topic name, so it will match namespaced/etc topic in Kafka # @note By default will not change topic (if default mapper used) mapped_topic = Karafka :: App . config . topic_mapper . outgoing ( options [ :topic ] ) external_options = options . merge ( topic : mapped_topic ) producer ( options ) . call ( data , external_options ) end end end


Input: This method allow us to respond to a single topic with a given data . It can be used as many times as we need . Especially when we have 1 : n flow [CODESPLIT] def respond_to ( topic , data , options = { } ) # We normalize the format to string, as WaterDrop and Ruby-Kafka support only # string topics topic = topic . to_s messages_buffer [ topic ] ||= [ ] messages_buffer [ topic ] << [ self . class . topics [ topic ] . serializer . call ( data ) , options . merge ( topic : topic ) ] end


Input: Informs monitoring about trapped signal [CODESPLIT] def notice_signal ( signal ) Thread . new do Karafka . monitor . instrument ( 'process.notice_signal' , caller : self , signal : signal ) end end


Input: Looks up ids in a list of embedding tensors . [CODESPLIT] def embedding_lookup ( params , ids , partition_strategy : "mod" , name : nil , validate_indices : true , max_norm : nil ) _embedding_lookup_and_transform ( params , ids , partition_strategy : partition_strategy , name : name , max_norm : max_norm , transform_fn : nil ) end


Input: Helper function for embedding_lookup and _compute_sampled_logits . [CODESPLIT] def _embedding_lookup_and_transform ( params , ids , partition_strategy : "mod" , name : nil , max_norm : nil , transform_fn : nil ) raise TensorStream :: ValueError , "Need at least one param" if params . nil? params = [ params ] unless params . is_a? ( Array ) TensorStream . name_scope ( name , "embedding_lookup" , values : params + [ ids ] ) do | name | np = params . size ids = TensorStream . convert_to_tensor ( ids , name : "ids" ) if ( np == 1 ) && ( transform_fn . nil? || ( ids . shape . size == 1 ) ) result = nil TensorStream . colocate_with ( params [ 0 ] ) do result = _clip ( TensorStream . gather ( params [ 0 ] , ids , name : name ) , ids , max_norm ) result = transform_fn . call ( result ) if transform_fn end return TensorStream . identity ( result ) else flat_ids = TensorStream . reshape ( ids , [ - 1 ] ) original_indices = TensorStream . range ( TensorStream . size ( flat_ids ) ) p_assignments = nil new_ids = nil if partition_strategy == "mod" p_assignments = flat_ids % np new_ids = floor_div ( flat_ids , np ) elsif partition_strategy == "div" raise "not yet supported!" else raise TensorStream :: ValueError , "Unrecognized partition strategy: " + partition_strategy end p_assignments = TensorStream . cast ( p_assignments , :int32 ) gather_ids = TensorStream . dynamic_partition ( new_ids , p_assignments , np ) pindices = TensorStream . dynamic_partition ( original_indices , p_assignments , np ) partitioned_result = [ ] ( 0 ... np ) . each do | p | pids = gather_ids [ p ] result = nil TensorStream . colocate_with ( params [ p ] ) do result = TensorStream . gather ( params [ p ] , pids ) if transform_fn # If transform_fn is provided, the clip_by_norm precedes # the transform and hence must be co-located. See below # for the counterpart if transform_fn is not proveded. result = transform_fn . call ( _clip ( result , pids , max_norm ) ) end end partitioned_result << result end ret = TensorStream . dynamic_stitch ( pindices , partitioned_result , name : name ) if transform_fn . nil? element_shape_s = params [ 0 ] . shape [ 1 .. - 1 ] params [ 1 .. - 1 ] . each { | p | element_shape_s = element_shape_s . merge_with ( p . shape [ 1 .. - 1 ] ) } else element_shape_s = ret . shape [ 1 .. - 1 ] end # Compute the dynamic element shape. element_shape_d = if element_shape_s . fully_defined? element_shape_s elsif transform_fn . nil? # It's important that we compute params[0].shape on the right device # to avoid data motion. TensorStream . colocate_with ( params [ 0 ] ) do params_shape = TensorStream . shape ( params [ 0 ] ) params_shape [ 1 .. - 1 ] end else TensorStream . shape ( ret ) [ 1 .. - 1 ] end ret = TensorStream . reshape ( ret , TensorStream . concat ( [ TensorStream . shape ( ids ) , element_shape_d ] , 0 ) ) ret = _clip ( ret , ids , max_norm ) unless transform_fn ret end end end


Input: parsers a protobuf file and spits out a ruby hash [CODESPLIT] def load ( pbfile ) f = File . new ( pbfile , "r" ) lines = [ ] while ! f . eof? && ( str = f . readline . strip ) lines << str end evaluate_lines ( lines ) end


Input: Assert the condition x == y holds element - wise . [CODESPLIT] def assert_equal ( x , y , data : nil , summarize : nil , message : nil , name : nil ) _op ( :assert_equal , x , y , data : data , summarize : summarize , message : message , name : name ) end


Input: Constructs symbolic derivatives of ys of input w . r . t . x in wrt_xs . [CODESPLIT] def gradients ( tensor_ys , wrt_xs , name : "gradients" , stop_gradients : nil ) tensor_ys = tensor_ys . op gs = wrt_xs . map ( :op ) . collect { | x | stops = stop_gradients ? stop_gradients . map ( :name ) . join ( "_" ) : "" gradient_program_name = "grad_#{tensor_ys.name}_#{x.name}_#{stops}" . to_sym tensor_graph = tensor_ys . graph tensor_program = if tensor_graph . node_added? ( gradient_program_name ) tensor_graph . get_node ( gradient_program_name ) else tensor_graph . name_scope ( "gradient_wrt_#{x.name}" ) do derivative_ops = TensorStream :: MathGradients . derivative ( tensor_ys , x , graph : tensor_graph , stop_gradients : stop_gradients ) tensor_graph . add_node! ( gradient_program_name , derivative_ops ) end end tensor_program } gs end


Input: Outputs random values from a normal distribution . [CODESPLIT] def random_normal ( shape , dtype : :float32 , mean : 0.0 , stddev : 1.0 , seed : nil , name : nil ) options = { dtype : dtype , mean : mean , stddev : stddev , seed : seed , name : name } _op ( :random_standard_normal , shape , options ) end


Input: Construct an identity matrix [CODESPLIT] def eye ( num_rows , num_columns : nil , dtype : :float32 , name : nil ) _op ( :eye , num_rows , num_columns || num_rows , data_type : dtype , name : name ) end


Input: The Glorot uniform initializer also called Xavier uniform initializer . [CODESPLIT] def glorot_uniform_initializer ( seed : nil , dtype : nil ) TensorStream :: Initializer . new ( -> { _op ( :glorot_uniform , seed : seed , data_type : dtype ) } ) end


Input: Initializer that generates tensors with a uniform distribution . [CODESPLIT] def random_uniform_initializer ( minval : 0 , maxval : 1 , seed : nil , dtype : nil ) TensorStream :: Initializer . new ( -> { _op ( :random_uniform , minval : 0 , maxval : 1 , seed : seed , data_type : dtype ) } ) end


Input: Extracts a slice from a tensor . [CODESPLIT] def slice ( input , start , size , name : nil ) _op ( :slice , input , start , size : size , name : name ) end


Input: Creates a tensor with all elements set to 1 . [CODESPLIT] def ones ( shape , dtype : :float32 , name : nil ) _op ( :ones , shape , data_type : dtype , name : name ) end


Input: Returns the truth value of x AND y element - wise . [CODESPLIT] def logical_and ( input_a , input_b , name : nil ) check_data_types ( input_a , input_b ) _op ( :logical_and , input_a , input_b , name : name ) end


Input: Computes the mean of elements across dimensions of a tensor . [CODESPLIT] def reduce_mean ( input_tensor , axis = nil , keepdims : false , name : nil ) reduce ( :mean , input_tensor , axis , keepdims : keepdims , name : name ) end


Input: Concatenates tensors along one dimension . [CODESPLIT] def concat ( values , axis , name : "concat" ) if values . is_a? ( Array ) _op ( :concat , axis , values , name : name ) else _op ( :concat , axis , values , name : name ) end end


Input: Partitions data into num_partitions tensors using indices from partitions [CODESPLIT] def dynamic_partition ( data , partitions , num_partitions , name : nil ) result = _op ( :dynamic_partition , data , partitions , num_partitions : num_partitions , name : nil ) num_partitions . times . map do | index | result [ index ] end end


Input: Return true_fn () if the predicate pred is true else false_fn () . [CODESPLIT] def cond ( pred , true_fn , false_fn , name : nil ) _op ( :case , [ pred ] , false_fn , true_fn , name : name ) end


Input: Return the elements either from x or y depending on the condition . [CODESPLIT] def where ( condition , true_t = nil , false_t = nil , name : nil ) _op ( :where , condition , true_t , false_t , name : name ) end


Input: Computes asin of input element - wise [CODESPLIT] def asin ( input , name : nil ) check_allowed_types ( input , FLOATING_POINT_TYPES ) _op ( :asin , input , name : name ) end


Input: Computes acos of input element - wise [CODESPLIT] def acos ( input , name : nil ) check_allowed_types ( input , FLOATING_POINT_TYPES ) _op ( :acos , input , name : name ) end


Input: Computes atan of input element - wise [CODESPLIT] def atan ( input , name : nil ) check_allowed_types ( input , FLOATING_POINT_TYPES ) _op ( :atan , input , name : name ) end


Input: Returns element - wise integer divistion . [CODESPLIT] def floor_div ( input_a , input_b , name : nil ) check_data_types ( input_a , input_b ) _op ( :floor_div , input_a , input_b , name : name ) end


Input: Casts a tensor to a new type if needed [CODESPLIT] def cast ( input , dtype , name : nil ) input = convert_to_tensor ( input ) return input if input . data_type == dtype _op ( :cast , input , data_type : dtype , name : name ) end


Input: Prints a list of tensors . [CODESPLIT] def print ( input , data , message : nil , name : nil ) _op ( :print , input , data , message : message , name : name ) end


Input: Returns the truth value of ( x ! = y ) element - wise . This ops supports broadcasting [CODESPLIT] def not_equal ( input_a , input_b , name : nil ) check_data_types ( input_a , input_b ) _op ( :not_equal , input_a , input_b , name : name ) end


Input: reates a tensor with all elements set to zero . Given a single tensor ( tensor ) this operation returns a tensor of the same type and shape as tensor with all elements set to zero . Optionally you can use dtype to specify a new type for the returned tensor . [CODESPLIT] def zeros_like ( tensor , dtype : nil , name : nil ) _op ( :zeros_like , tensor , data_type : dtype , name : name ) end


Input: Creates a tensor with all elements set to 1 . Given a single tensor ( tensor ) this operation returns a tensor of the same type and shape as tensor with all elements set to 1 . Optionally you can specify a new type ( dtype ) for the returned tensor . [CODESPLIT] def ones_like ( tensor , dtype : nil , name : nil ) _op ( :ones_like , tensor , data_type : dtype , name : name ) end


Input: Returns x * y element - wise . This operation supports broadcasting [CODESPLIT] def multiply ( input_a , input_b , name : nil ) check_data_types ( input_a , input_b ) _op ( :mul , input_a , input_b , name : name ) end


Input: Computes sec of input element - wise . [CODESPLIT] def sec ( input , name : nil ) check_allowed_types ( input , FLOATING_POINT_TYPES ) _op ( :sec , input , name : name ) end


Input: Computes sqrt of input element - wise . [CODESPLIT] def sqrt ( input , name : nil ) check_allowed_types ( input , FLOATING_POINT_TYPES ) _op ( :sqrt , input , name : name ) end


Input: Computes natural logarithm of x element - wise . [CODESPLIT] def log ( input , name : nil ) check_allowed_types ( input , FLOATING_POINT_TYPES ) _op ( :log , input , name : name ) end


Input: Computes natural logarithm of ( 1 + x ) element - wise . [CODESPLIT] def log1p ( input , name : nil ) check_allowed_types ( input , FLOATING_POINT_TYPES ) _op ( :log1p , input , name : name ) end


Input: Computes exponential of x element - wise . [CODESPLIT] def exp ( input , name : nil ) check_allowed_types ( input , FLOATING_POINT_TYPES ) _op ( :exp , input , name : name ) end


Input: Pads a tensor . This operation pads a tensor according to the paddings you specify . [CODESPLIT] def pad ( tensor , paddings , mode : "CONSTANT" , name : nil ) _op ( :pad , tensor , paddings , mode : mode , name : name ) end


Input: Checks a tensor for NaN and Inf values . When run reports an InvalidArgument error if tensor has any values that are not a number ( NaN ) or infinity ( Inf ) . Otherwise passes tensor as - is . [CODESPLIT] def check_numerics ( tensor , message , name : nil ) _op ( :check_numerics , tensor , message : message , name : name ) end


Input: Gather slices from params and axis according to indices . [CODESPLIT] def gather ( params , indices , validate_indices : nil , name : nil , axis : 0 ) _op ( :gather , params , indices , validate_indices : validate_indices , name : name , axis : axis ) end


Input: Stacks a list of rank - R tensors into one rank - ( R + 1 ) tensor . [CODESPLIT] def stack ( values , axis : 0 , name : "stack" ) _op ( :stack , values , axis : axis , name : name ) end


Input: Unpacks the given dimension of a rank - R tensor into rank - ( R - 1 ) tensors . [CODESPLIT] def unstack ( value , num : nil , axis : 0 , name : "unstack" ) res = _op ( :unstack , value , num : num , axis : axis , name : name ) num_vars = if value . shape . known? new_shape = value . shape . shape . dup rank = new_shape . size - 1 axis = rank + axis if axis < 0 rotated_shape = Array . new ( axis + 1 ) { new_shape . shift } new_shape = rotated_shape . rotate! ( - 1 ) + new_shape new_shape [ 0 ] else raise TensorStream :: ValueError , "num is unspecified and cannot be inferred." if num . nil? num end return res [ 0 ] if num_vars == 1 Array . new ( num_vars ) do | i | index ( res , i , name : "unstack/index:#{i}" ) end end


Input: Same as stack [CODESPLIT] def pack ( values , axis : 0 , name : "pack" ) _op ( :stack , values , axis : axis , name : name ) end


Input: Same as unstack [CODESPLIT] def unpack ( value , num : nil , axis : 0 , name : "unpack" ) unstack ( value , num : num , axis : axis , name : name ) end


Input: Computes the difference between two lists of numbers or strings . Given a list x and a list y this operation returns a list out that represents all values that are in x but not in y . The returned list out is sorted in the same order that the numbers appear in x ( duplicates are preserved ) . This operation also returns a list idx that represents the position of each out element in x . In other words : [CODESPLIT] def setdiff1d ( x , y , index_dtype : :int32 , name : nil ) result = _op ( :setdiff1d , x , y , index_dtype : index_dtype , name : name ) [ result [ 0 ] , result [ 1 ] ] end


Input: Create a case operation . [CODESPLIT] def case ( args = { } ) args = args . dup default = args . delete ( :default ) exclusive = args . delete ( :exclusive ) strict = args . delete ( :strict ) name = args . delete ( :name ) predicates = [ ] functions = [ ] args . each do | k , v | raise "Invalid argment or option #{k}" unless k . is_a? ( Tensor ) predicates << k functions << ( v . is_a? ( Proc ) ? v . call : v ) end _op ( :case , predicates , default , functions , exclusive : exclusive , strict : strict , name : name ) end


Input: same as op but with a marker that it was internal generated [CODESPLIT] def i_op ( code , * args ) options = if args . last . is_a? ( Hash ) args . pop else { } end args << options . merge ( internal : true ) Graph . get_default_graph . add_op! ( code . to_sym , args ) end


Input: explicit broadcasting helper [CODESPLIT] def broadcast_dimensions ( input , dims = [ ] ) return input if dims . empty? d = dims . shift if input . is_a? ( Array ) && ( get_rank ( input ) - 1 ) == dims . size row_to_dup = input . collect { | item | broadcast_dimensions ( item , dims . dup ) } row_to_dup + Array . new ( d ) { row_to_dup } . flatten ( 1 ) elsif input . is_a? ( Array ) Array . new ( d ) { broadcast_dimensions ( input , dims . dup ) } else Array . new ( d + 1 ) { input } end end


Input: handle 2 tensor math operations [CODESPLIT] def vector_op ( vector , vector2 , switch = false , safe = true , & block ) if get_rank ( vector ) < get_rank ( vector2 ) # upgrade rank of A duplicated = Array . new ( vector2 . size ) { vector } return vector_op ( duplicated , vector2 , switch , block ) end return yield ( vector , vector2 ) unless vector . is_a? ( Array ) vector . each_with_index . collect { | input , index | next vector_op ( input , vector2 , switch , block ) if input . is_a? ( Array ) && get_rank ( vector ) > get_rank ( vector2 ) if safe && vector2 . is_a? ( Array ) next nil if vector2 . size != 1 && index >= vector2 . size end z = if vector2 . is_a? ( Array ) if index < vector2 . size vector2 [ index ] else raise "incompatible tensor shapes used during op" if vector2 . size != 1 vector2 [ 0 ] end else vector2 end if input . is_a? ( Array ) vector_op ( input , z , switch , block ) else switch ? yield ( z , input ) : yield ( input , z ) end } . compact end


Input: general case transposition with flat arrays [CODESPLIT] def transpose_with_perm ( arr , new_arr , shape , new_shape , perm ) arr_size = shape . reduce ( :* ) divisors = shape . dup . drop ( 1 ) . reverse . inject ( [ 1 ] ) { | a , s | a << s * a . last } . reverse multipliers = new_shape . dup . drop ( 1 ) . reverse . inject ( [ 1 ] ) { | a , s | a << s * a . last } . reverse arr_size . times do | p | ptr = p index = [ ] divisors . each_with_object ( index ) do | div , a | a << ( ptr / div . to_f ) . floor ptr = ptr % div end # remap based on perm remaped = perm . map { | x | index [ x ] } ptr2 = 0 multipliers . each_with_index do | m , idx | ptr2 += remaped [ idx ] * m end new_arr [ ptr2 ] = arr [ p ] end [ new_arr , new_shape ] end


Input: Returns x + y element - wise . [CODESPLIT] def add ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :add , input_a , input_b , name : name ) end


Input: Returns the index with the largest value across axes of a tensor . [CODESPLIT] def argmax ( input_a , axis = nil , name : nil , dimension : nil , output_type : :int32 ) check_allowed_types ( input_a , TensorStream :: Ops :: NUMERIC_TYPES ) check_allowed_types ( axis , TensorStream :: Ops :: INTEGER_TYPES ) _op ( :argmax , input_a , axis , name : name , dimension : dimension , output_type : output_type ) end


Input: Returns element - wise smallest integer in not less than x [CODESPLIT] def ceil ( input_a , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: FLOATING_POINT_TYPES ) _op ( :ceil , input_a , name : name ) end


Input: Computes cos of input element - wise . [CODESPLIT] def cos ( input_a , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: FLOATING_POINT_TYPES ) _op ( :cos , input_a , name : name ) end


Input: Returns x / y element - wise . [CODESPLIT] def div ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :div , input_a , input_b , name : name ) end


Input: Returns the truth value of ( x == y ) element - wise . [CODESPLIT] def equal ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :equal , input_a , input_b , name : name ) end


Input: Returns element - wise largest integer not greater than x . [CODESPLIT] def floor ( input_a , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: FLOATING_POINT_TYPES ) _op ( :floor , input_a , name : name ) end


Input: Returns element - wise integer divistion . [CODESPLIT] def floor_div ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :floor_div , input_a , input_b , name : name ) end


Input: Returns the truth value of ( x > y ) element - wise . [CODESPLIT] def greater ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :greater , input_a , input_b , name : name ) end


Input: Returns the truth value of ( x > = y ) element - wise . [CODESPLIT] def greater_equal ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :greater_equal , input_a , input_b , name : name ) end


Input: Returns the truth value of ( x < y ) element - wise . [CODESPLIT] def less ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :less , input_a , input_b , name : name ) end


Input: Returns the truth value of ( x < = y ) element - wise . [CODESPLIT] def less_equal ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :less_equal , input_a , input_b , name : name ) end


Input: Multiplies matrix a by matrix b producing a * b . The inputs must following any transpositions be tensors of rank 2 . [CODESPLIT] def mat_mul ( input_a , input_b , transpose_a : false , transpose_b : false , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :mat_mul , input_a , input_b , transpose_a : transpose_a , transpose_b : transpose_b , name : name ) end


Input: Returns the max of x and y ( i . e . x > y ? x : y ) element - wise . [CODESPLIT] def max ( input_a , input_b , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: NUMERIC_TYPES ) check_allowed_types ( input_b , TensorStream :: Ops :: NUMERIC_TYPES ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :max , input_a , input_b , name : name ) end


Input: Returns element - wise remainder of division . [CODESPLIT] def mod ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :mod , input_a , input_b , name : name ) end


Input: Returns x * y element - wise . [CODESPLIT] def mul ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :mul , input_a , input_b , name : name ) end


Input: Returns the truth value of ( x ! = y ) element - wise . [CODESPLIT] def not_equal ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :not_equal , input_a , input_b , name : name ) end


Input: Creates a tensor with all elements set to 1 . Given a single tensor ( tensor ) this operation returns a tensor of the same type and shape as tensor with all elements set to 1 . Optionally you can specify a new type ( dtype ) for the returned tensor . [CODESPLIT] def ones_like ( input , dtype : nil , name : nil ) _op ( :ones_like , input , data_type : dtype , name : name ) end


Input: Computes the power of one value to another X^Y element wise [CODESPLIT] def pow ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :pow , input_a , input_b , name : name ) end


Input: Computes the product of elements across dimensions of a tensor . Reduces input_tensor along the dimensions given in axis . Unless keepdims is true the rank of the tensor is reduced by 1 for each entry in axis . If keepdims is true the reduced dimensions are retained with length 1 . If axis has no entries all dimensions are reduced and a tensor with a single element is returned . [CODESPLIT] def prod ( input_a , axis = nil , name : nil , keepdims : false ) check_allowed_types ( axis , TensorStream :: Ops :: INTEGER_TYPES ) input_a = TensorStream . convert_to_tensor ( input_a ) return input_a if input_a . shape . scalar? axis = cast_axis ( input_a , axis ) _op ( :prod , input_a , axis , name : name , keepdims : keepdims ) end


Input: Outputs random values from a uniform distribution . [CODESPLIT] def random_uniform ( shape , name : nil , dtype : :float32 , minval : 0 , maxval : 1 , seed : nil ) _op ( :random_uniform , shape , name : name , dtype : dtype , minval : minval , maxval : maxval , seed : seed ) end


Input: Creates a sequence of numbers . Creates a sequence of numbers that begins at start and extends by increments of delta up to but not including limit . [CODESPLIT] def range ( start = 0 , limit = 0 , delta = 1 , name : "range" , dtype : nil , output_type : :int32 ) _op ( :range , start , limit , delta , name : name , dtype : dtype , output_type : output_type ) end


Input: Returns the rank of a tensor [CODESPLIT] def rank ( input , name : nil ) input = convert_to_tensor ( input ) return cons ( input . shape . ndims ) if input . shape . known? _op ( :rank , input , name : name ) end


Input: Rounds the values of a tensor to the nearest integer element - wise [CODESPLIT] def round ( input_a , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: FLOATING_POINT_TYPES ) _op ( :round , input_a , name : name ) end


Input: Computes reciprocal of square root of x element - wise . [CODESPLIT] def rsqrt ( input_a , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: FLOATING_POINT_TYPES ) _op ( :rsqrt , input_a , name : name ) end


Input: This operation returns a 1 - D integer tensor representing the shape of input [CODESPLIT] def shape ( input , name : nil , out_type : :int32 ) return constant ( shape_eval ( input , out_type ) , dtype : out_type , name : "Shape/#{name}" ) if input . is_a? ( Array ) && ! input [ 0 ] . is_a? ( Tensor ) return constant ( input . shape . shape , dtype : out_type , name : "Shape/#{input.name}_c" ) if shape_full_specified ( input ) _op ( :shape , input , name : name , out_type : out_type ) end


Input: Computes sigmoid of x element - wise . [CODESPLIT] def sigmoid ( input_a , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: FLOATING_POINT_TYPES ) _op ( :sigmoid , input_a , name : name ) end


Input: Computes sin of input element - wise . [CODESPLIT] def sin ( input_a , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: FLOATING_POINT_TYPES ) _op ( :sin , input_a , name : name ) end


Input: Returns the size of a tensor . Returns a 0 - D Tensor representing the number of elements in input of type out_type . Defaults to : int32 . [CODESPLIT] def size ( input , name : nil , out_type : :int32 ) _op ( :size , input , name : name , out_type : out_type ) end


Input: Extracts a strided slice of a tensor this op extracts a slice of size ( end - begin ) / stride from the given input_ tensor . Starting at the location specified by begin the slice continues by adding stride to the index until all dimensions are not less than end . Note that a stride can be negative which causes a reverse slice . [CODESPLIT] def strided_slice ( input , _begin , _end , strides = nil , name : nil ) _op ( :strided_slice , input , _begin , _end , strides , name : name ) end


Input: Returns x - y element - wise . [CODESPLIT] def sub ( input_a , input_b , name : nil ) input_a , input_b = apply_data_type_coercion ( input_a , input_b ) _op ( :sub , input_a , input_b , name : name ) end


Input: Computes the sum of elements across dimensions of a tensor . Reduces input_tensor along the dimensions given in axis . Unless keepdims is true the rank of the tensor is reduced by 1 for each entry in axis . If keepdims is true the reduced dimensions are retained with length 1 . If axis has no entries all dimensions are reduced and a tensor with a single element is returned . [CODESPLIT] def sum ( input_a , axis_p = nil , axis : nil , name : nil , keepdims : false ) check_allowed_types ( axis_p , TensorStream :: Ops :: INTEGER_TYPES ) input_a = TensorStream . convert_to_tensor ( input_a ) return input_a if input_a . shape . scalar? axis_p = axis_p || axis axis_p = cast_axis ( input_a , axis_p ) _op ( :sum , input_a , axis_p , name : name , keepdims : keepdims ) end


Input: Computes tan of input element - wise . [CODESPLIT] def tan ( input_a , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: FLOATING_POINT_TYPES ) _op ( :tan , input_a , name : name ) end


Input: Computes tanh of input element - wise . [CODESPLIT] def tanh ( input_a , name : nil ) check_allowed_types ( input_a , TensorStream :: Ops :: FLOATING_POINT_TYPES ) _op ( :tanh , input_a , name : name ) end


Input: Finds values and indices of the k largest entries for the last dimension . [CODESPLIT] def top_k ( input , k = 1 , sorted : true , name : nil ) result = _op ( :top_k , input , k , sorted : sorted , name : name ) [ result [ 0 ] , result [ 1 ] ] end


Input: Creates a tensor with all elements set to zero [CODESPLIT] def zeros ( shape , dtype : :float32 , name : nil ) _op ( :zeros , shape , dtype : dtype , name : name ) end


Input: Utility class to convert variables to constants for production deployment [CODESPLIT] def convert ( session , checkpoint_folder , output_file ) model_file = File . join ( checkpoint_folder , "model.yaml" ) TensorStream . graph . as_default do | current_graph | YamlLoader . new . load_from_string ( File . read ( model_file ) ) saver = TensorStream :: Train :: Saver . new saver . restore ( session , checkpoint_folder ) # collect all assign ops and remove them from the graph remove_nodes = Set . new ( current_graph . nodes . values . select { | op | op . is_a? ( TensorStream :: Operation ) && op . operation == :assign } . map { | op | op . consumers . to_a } . flatten . uniq ) output_buffer = TensorStream :: Yaml . new . get_string ( current_graph ) { | graph , node_key | node = graph . get_tensor_by_name ( node_key ) case node . operation when :variable_v2 value = node . container options = { value : value , data_type : node . data_type , shape : shape_eval ( value ) , } const_op = TensorStream :: Operation . new ( current_graph , inputs : [ ] , options : options ) const_op . name = node . name const_op . operation = :const const_op . data_type = node . data_type const_op . shape = TensorShape . new ( shape_eval ( value ) ) const_op when :assign nil else remove_nodes . include? ( node . name ) ? nil : node end } File . write ( output_file , output_buffer ) end end


Input: Returns a context manager that specifies the default device to use . [CODESPLIT] def device ( device_name ) Thread . current [ "ts_graph_#{object_id}" ] ||= { } Thread . current [ "ts_graph_#{object_id}" ] [ :default_device ] ||= [ ] Thread . current [ "ts_graph_#{object_id}" ] [ :default_device ] << device_name begin yield ensure Thread . current [ "ts_graph_#{object_id}" ] [ :default_device ] . pop end end


Input: Loads a model Yaml file and builds the model from it [CODESPLIT] def load_from_string ( buffer ) serialized_ops = YAML . safe_load ( buffer , [ Symbol ] , [ ] , true ) serialized_ops . each do | op_def | inputs = op_def [ :inputs ] . map { | i | @graph . get_tensor_by_name ( i ) } options = { } new_var = nil if op_def . dig ( :attrs , :container ) new_var = Variable . new ( op_def . dig ( :attrs , :data_type ) ) var_shape = op_def . dig ( :attrs , :container , :shape ) var_options = op_def . dig ( :attrs , :container , :options ) var_options [ :name ] = op_def [ :name ] new_var . prepare ( var_shape . size , var_shape , TensorStream . get_variable_scope , var_options ) options [ :container ] = new_var @graph . add_variable ( new_var , var_options ) end new_op = Operation . new ( @graph , inputs : inputs , options : op_def [ :attrs ] . merge ( options ) ) new_op . operation = op_def [ :op ] . to_sym new_op . name = op_def [ :name ] new_op . shape = TensorShape . new ( TensorStream :: InferShape . infer_shape ( new_op ) ) new_op . rank = new_op . shape . rank new_op . data_type = new_op . set_data_type ( op_def . dig ( :attrs , :data_type ) ) new_op . is_const = new_op . infer_const new_op . given_name = new_op . name new_var . op = new_op if new_var @graph . add_node ( new_op ) end @graph end