Method takeover has a Cognitive Complexity of 57 (exceeds 5 allowed). Consider refactoring.

Open

    def takeover
      @urls.each do |u|
        add_url(u) { |page| page.user_data.p_seeded = true }
      end
      return if internal_queue.empty?

• Read up
  Read up

File polipus.rb has 341 lines of code (exceeds 250 allowed). Consider refactoring.

Open

require 'redis'
require 'redis/connection/hiredis'
require 'redis-queue'
require 'polipus/version'
require 'polipus/http'

Class PolipusCrawler has 33 methods (exceeds 20 allowed). Consider refactoring.

Open

  class PolipusCrawler
    OPTS = {
      # run 4 threads
      workers: 4,
      # identify self as Polipus/VERSION

Method rotate has a Cognitive Complexity of 25 (exceeds 5 allowed). Consider refactoring.

Open

      def rotate(source, dest)
        performed = 0
        loop do
          message = source.pop(true)
          if message

• Read up
  Read up

Method takeover has 77 lines of code (exceeds 25 allowed). Consider refactoring.

Open

    def takeover
      @urls.each do |u|
        add_url(u) { |page| page.user_data.p_seeded = true }
      end
      return if internal_queue.empty?

Method get_response has a Cognitive Complexity of 17 (exceeds 5 allowed). Consider refactoring.

Open

    def get_response(url, referer = nil)
      full_path = url.query.nil? ? url.path : "#{url.path}?#{url.query}"

      opts = {}
      opts['User-Agent'] = user_agent if user_agent

• Read up
  Read up

Method initialize has a Cognitive Complexity of 17 (exceeds 5 allowed). Consider refactoring.

Open

      def initialize(uri, user_agent)
        io = Robotex.get_robots_txt(uri, user_agent)
        if !io || io.content_type != 'text/plain' || io.status != %w(200 OK)
          io = StringIO.new("User-agent: *\nAllow: /\n")
        end

• Read up
  Read up

Class HTTP has 21 methods (exceeds 20 allowed). Consider refactoring.

Open

  class HTTP
    # Maximum number of redirects to follow on each get_response
    REDIRECT_LIMIT = 5
    RESCUABLE_ERRORS = [
      EOFError,

Method initialize has a Cognitive Complexity of 16 (exceeds 5 allowed). Consider refactoring.

Open

    def initialize(job_name = 'polipus', urls = [], options = {})
      @job_name     = job_name
      @options      = OPTS.merge(options)
      @options[:queue_timeout] = 1 if @options[:queue_timeout] <= 0
      @logger       = @options[:logger]  ||= Logger.new(nil)

• Read up
  Read up

Method get has a Cognitive Complexity of 15 (exceeds 5 allowed). Consider refactoring.

Open

    def get(url, referer = nil)
      limit = redirect_limit
      loc = url
      loop do
        # if redirected to a relative url, merge it with the host of the original

• Read up
  Read up

Method initialize has 45 lines of code (exceeds 25 allowed). Consider refactoring.

Open

    def initialize(job_name = 'polipus', urls = [], options = {})
      @job_name     = job_name
      @options      = OPTS.merge(options)
      @options[:queue_timeout] = 1 if @options[:queue_timeout] <= 0
      @logger       = @options[:logger]  ||= Logger.new(nil)

Method links has a Cognitive Complexity of 14 (exceeds 5 allowed). Consider refactoring.

Open

    def links
      return @links.to_a unless @links.nil?
      @links = Set.new
      return [] unless doc

• Read up
  Read up

Method base has a Cognitive Complexity of 14 (exceeds 5 allowed). Consider refactoring.

Open

    def base
      @base = if doc
                href = doc.search('//head/base/@href')
                URI(href.to_s) unless href.nil? rescue nil
              end unless @base

• Read up
  Read up

Method from_hash has a Cognitive Complexity of 13 (exceeds 5 allowed). Consider refactoring.

Open

    def self.from_hash(hash)
      page = new(URI(hash['url']))
      {
        '@headers'       => hash['headers'] && !hash['headers'].empty? ? Marshal.load(hash['headers']) : { 'content-type' => [''] },
        '@body'          => hash['body'],

• Read up
  Read up

Method initialize has 32 lines of code (exceeds 25 allowed). Consider refactoring.

Open

      def initialize(uri, user_agent)
        io = Robotex.get_robots_txt(uri, user_agent)
        if !io || io.content_type != 'text/plain' || io.status != %w(200 OK)
          io = StringIO.new("User-agent: *\nAllow: /\n")
        end

Method get_response has 30 lines of code (exceeds 25 allowed). Consider refactoring.

Open

    def get_response(url, referer = nil)
      full_path = url.query.nil? ? url.path : "#{url.path}?#{url.query}"

      opts = {}
      opts['User-Agent'] = user_agent if user_agent

Method fetch_pages has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring.

Open

    def fetch_pages(url, referer = nil, depth = nil, user_data = nil)
      url = URI(url)
      pages = []
      get(url, referer) do |response, code, location, redirect_to, response_time|
        handle_compression response

• Read up
  Read up

Method allowed? has a Cognitive Complexity of 10 (exceeds 5 allowed). Consider refactoring.

Open

      def allowed?(uri, user_agent)
        return true unless @parsed
        allowed = true
        uri = URI.parse(uri.to_s) unless uri.is_a?(URI)
        path = uri.request_uri

• Read up
  Read up

Method connection has a Cognitive Complexity of 9 (exceeds 5 allowed). Consider refactoring.

Open

    def connection(url)
      @connections[url.host] ||= {}
      @connections_hits[url.host] ||= {}

      if @connections[url.host][url.port]

• Read up
  Read up

Method add has a Cognitive Complexity of 8 (exceeds 5 allowed). Consider refactoring.

Open

      def add(page)
        @semaphore.synchronize do
          obj = page.to_hash
          @except.each { |e| obj.delete e.to_s }
          obj[:id] = uuid(page)

• Read up
  Read up