Showing 28 of 28 total issues
Method takeover
has a Cognitive Complexity of 57 (exceeds 5 allowed). Consider refactoring. Open
def takeover
@urls.each do |u|
add_url(u) { |page| page.user_data.p_seeded = true }
end
return if internal_queue.empty?
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
File polipus.rb
has 341 lines of code (exceeds 250 allowed). Consider refactoring. Open
require 'redis'
require 'redis/connection/hiredis'
require 'redis-queue'
require 'polipus/version'
require 'polipus/http'
Class PolipusCrawler
has 33 methods (exceeds 20 allowed). Consider refactoring. Open
class PolipusCrawler
OPTS = {
# run 4 threads
workers: 4,
# identify self as Polipus/VERSION
Method rotate
has a Cognitive Complexity of 25 (exceeds 5 allowed). Consider refactoring. Open
def rotate(source, dest)
performed = 0
loop do
message = source.pop(true)
if message
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method takeover
has 77 lines of code (exceeds 25 allowed). Consider refactoring. Open
def takeover
@urls.each do |u|
add_url(u) { |page| page.user_data.p_seeded = true }
end
return if internal_queue.empty?
Method initialize
has a Cognitive Complexity of 17 (exceeds 5 allowed). Consider refactoring. Open
def initialize(uri, user_agent)
io = Robotex.get_robots_txt(uri, user_agent)
if !io || io.content_type != 'text/plain' || io.status != %w(200 OK)
io = StringIO.new("User-agent: *\nAllow: /\n")
end
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method get_response
has a Cognitive Complexity of 17 (exceeds 5 allowed). Consider refactoring. Open
def get_response(url, referer = nil)
full_path = url.query.nil? ? url.path : "#{url.path}?#{url.query}"
opts = {}
opts['User-Agent'] = user_agent if user_agent
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Class HTTP
has 21 methods (exceeds 20 allowed). Consider refactoring. Open
class HTTP
# Maximum number of redirects to follow on each get_response
REDIRECT_LIMIT = 5
RESCUABLE_ERRORS = [
EOFError,
Method initialize
has a Cognitive Complexity of 16 (exceeds 5 allowed). Consider refactoring. Open
def initialize(job_name = 'polipus', urls = [], options = {})
@job_name = job_name
@options = OPTS.merge(options)
@options[:queue_timeout] = 1 if @options[:queue_timeout] <= 0
@logger = @options[:logger] ||= Logger.new(nil)
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method get
has a Cognitive Complexity of 15 (exceeds 5 allowed). Consider refactoring. Open
def get(url, referer = nil)
limit = redirect_limit
loc = url
loop do
# if redirected to a relative url, merge it with the host of the original
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method initialize
has 45 lines of code (exceeds 25 allowed). Consider refactoring. Open
def initialize(job_name = 'polipus', urls = [], options = {})
@job_name = job_name
@options = OPTS.merge(options)
@options[:queue_timeout] = 1 if @options[:queue_timeout] <= 0
@logger = @options[:logger] ||= Logger.new(nil)
Method links
has a Cognitive Complexity of 14 (exceeds 5 allowed). Consider refactoring. Open
def links
return @links.to_a unless @links.nil?
@links = Set.new
return [] unless doc
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method base
has a Cognitive Complexity of 14 (exceeds 5 allowed). Consider refactoring. Open
def base
@base = if doc
href = doc.search('//head/base/@href')
URI(href.to_s) unless href.nil? rescue nil
end unless @base
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method from_hash
has a Cognitive Complexity of 13 (exceeds 5 allowed). Consider refactoring. Open
def self.from_hash(hash)
page = new(URI(hash['url']))
{
'@headers' => hash['headers'] && !hash['headers'].empty? ? Marshal.load(hash['headers']) : { 'content-type' => [''] },
'@body' => hash['body'],
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method initialize
has 32 lines of code (exceeds 25 allowed). Consider refactoring. Open
def initialize(uri, user_agent)
io = Robotex.get_robots_txt(uri, user_agent)
if !io || io.content_type != 'text/plain' || io.status != %w(200 OK)
io = StringIO.new("User-agent: *\nAllow: /\n")
end
Method get_response
has 30 lines of code (exceeds 25 allowed). Consider refactoring. Open
def get_response(url, referer = nil)
full_path = url.query.nil? ? url.path : "#{url.path}?#{url.query}"
opts = {}
opts['User-Agent'] = user_agent if user_agent
Method allowed?
has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring. Open
def allowed?(uri, user_agent)
return true unless @parsed
allowed = true
uri = URI.parse(uri.to_s) unless uri.is_a?(URI)
path = uri.request_uri
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method fetch_pages
has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring. Open
def fetch_pages(url, referer = nil, depth = nil, user_data = nil)
url = URI(url)
pages = []
get(url, referer) do |response, code, location, redirect_to, response_time|
handle_compression response
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method connection
has a Cognitive Complexity of 9 (exceeds 5 allowed). Consider refactoring. Open
def connection(url)
@connections[url.host] ||= {}
@connections_hits[url.host] ||= {}
if @connections[url.host][url.port]
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"
Further reading
Method add
has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring. Open
def add(page)
@semaphore.synchronize do
obj = page.to_hash
@except.each { |e| obj.delete e.to_s }
obj[:id] = uuid(page)
- Read upRead up
Cognitive Complexity
Cognitive Complexity is a measure of how difficult a unit of code is to intuitively understand. Unlike Cyclomatic Complexity, which determines how difficult your code will be to test, Cognitive Complexity tells you how difficult your code will be to read and comprehend.
A method's cognitive complexity is based on a few simple rules:
- Code is not considered more complex when it uses shorthand that the language provides for collapsing multiple statements into one
- Code is considered more complex for each "break in the linear flow of the code"
- Code is considered more complex when "flow breaking structures are nested"