Sign upLogin

LucaCappelletti94/tinycrawler

View on GitHub
Star
Tweaked parser #317
LucaCappelletti94 authored a8d22740
complete00:00:37
Your first build completed successfully!
See the results

Congratulations

Your first build completed successfully!

See the Results

1
git clone
2
codeclimate validate-config
View output
codeclimate validate-config
No errors or warnings found in .codeclimate.json.
3
codeclimate prepare
View output
codeclimate prepare
4
builder pull-engines
View output
determining required images
docker pull codeclimate/codeclimate-apexmetrics:b512
b512: Pulling from codeclimate/codeclimate-apexmetrics
5040bd298390: Pulling fs layer
fce5728aad85: Pulling fs layer
c42794440453: Pulling fs layer
0c0da797ba48: Pulling fs layer
7c9b17433752: Pulling fs layer
114e02586e63: Pulling fs layer
e4c663802e9a: Pulling fs layer
63655a090396: Pulling fs layer
f54d2bb1ccfc: Pulling fs layer
d62e88f61c51: Pulling fs layer
c716404315a2: Pulling fs layer
e4c663802e9a: Waiting
63655a090396: Waiting
f54d2bb1ccfc: Waiting
d62e88f61c51: Waiting
c716404315a2: Waiting
0c0da797ba48: Waiting
7c9b17433752: Waiting
114e02586e63: Waiting
c42794440453: Verifying Checksum
c42794440453: Download complete
0c0da797ba48: Verifying Checksum
0c0da797ba48: Download complete
7c9b17433752: Download complete
fce5728aad85: Verifying Checksum
fce5728aad85: Download complete
e4c663802e9a: Download complete
5040bd298390: Download complete
f54d2bb1ccfc: Download complete
d62e88f61c51: Verifying Checksum
d62e88f61c51: Download complete
114e02586e63: Verifying Checksum
114e02586e63: Download complete
63655a090396: Verifying Checksum
63655a090396: Download complete
5040bd298390: Pull complete
c716404315a2: Verifying Checksum
c716404315a2: Download complete
fce5728aad85: Pull complete
c42794440453: Pull complete
0c0da797ba48: Pull complete
7c9b17433752: Pull complete
114e02586e63: Pull complete
e4c663802e9a: Pull complete
63655a090396: Pull complete
f54d2bb1ccfc: Pull complete
d62e88f61c51: Pull complete
c716404315a2: Pull complete
Digest: sha256:8e0f87ce33fec849507b10ad15b5ca1f165fb10e85bed96b8a71440b555ba16a
Status: Downloaded newer image for registry.prod.codeclimate.net/codeclimate/codeclimate-apexmetrics:b512
docker pull codeclimate/codeclimate-bandit:b866
b866: Pulling from codeclimate/codeclimate-bandit
a0e0ef996968: Pulling fs layer
00c69949e211: Pulling fs layer
124b11ffd9dc: Pulling fs layer
74d262bd6915: Pulling fs layer
963a4e0e1415: Pulling fs layer
ea041886215a: Pulling fs layer
3bbf7ff90c63: Pulling fs layer
963a4e0e1415: Waiting
74d262bd6915: Waiting
ea041886215a: Waiting
00c69949e211: Download complete
a0e0ef996968: Verifying Checksum
a0e0ef996968: Download complete
124b11ffd9dc: Verifying Checksum
124b11ffd9dc: Download complete
a0e0ef996968: Pull complete
963a4e0e1415: Verifying Checksum
963a4e0e1415: Download complete
ea041886215a: Verifying Checksum
ea041886215a: Download complete
00c69949e211: Pull complete
3bbf7ff90c63: Verifying Checksum
3bbf7ff90c63: Download complete
124b11ffd9dc: Pull complete
74d262bd6915: Verifying Checksum
74d262bd6915: Download complete
74d262bd6915: Pull complete
963a4e0e1415: Pull complete
ea041886215a: Pull complete
3bbf7ff90c63: Pull complete
Digest: sha256:b831294279e697e4402c0bd94153247577b1e1297055bac81a8e2922b4dedba5
Status: Downloaded newer image for registry.prod.codeclimate.net/codeclimate/codeclimate-bandit:b866
docker pull codeclimate/codeclimate-vint:b186
b186: Pulling from codeclimate/codeclimate-vint
f58436d1df4c: Pulling fs layer
a3ed95caeb02: Pulling fs layer
5729e645edd8: Pulling fs layer
66c1497ed962: Pulling fs layer
2b44cf65bb22: Pulling fs layer
efa53b2613f9: Pulling fs layer
841148c55afe: Pulling fs layer
b8e30729d5cd: Pulling fs layer
5729e645edd8: Waiting
2b44cf65bb22: Waiting
efa53b2613f9: Waiting
841148c55afe: Waiting
66c1497ed962: Waiting
b8e30729d5cd: Waiting
a3ed95caeb02: Verifying Checksum
a3ed95caeb02: Download complete
f58436d1df4c: Verifying Checksum
66c1497ed962: Verifying Checksum
66c1497ed962: Download complete
f58436d1df4c: Pull complete
2b44cf65bb22: Verifying Checksum
2b44cf65bb22: Download complete
5729e645edd8: Verifying Checksum
5729e645edd8: Download complete
a3ed95caeb02: Pull complete
841148c55afe: Verifying Checksum
841148c55afe: Download complete
efa53b2613f9: Verifying Checksum
efa53b2613f9: Download complete
b8e30729d5cd: Verifying Checksum
b8e30729d5cd: Download complete
5729e645edd8: Pull complete
66c1497ed962: Pull complete
2b44cf65bb22: Pull complete
efa53b2613f9: Pull complete
841148c55afe: Pull complete
b8e30729d5cd: Pull complete
Digest: sha256:134df01882fc111899aedb027979658d5e801101254d64a2c60daa1d18ba72a8
Status: Downloaded newer image for registry.prod.codeclimate.net/codeclimate/codeclimate-vint:b186
5
structure
View output
12
Parser process id: 12
codeclimate-parser socket not present
waiting 1s...
6
duplication
View output
12
Parser process id: 12
codeclimate-parser socket not present
waiting 1s...
I, [2024-04-08T07:45:13.616699 #1]  INFO -- : Skipping file ./tinycrawler/eta/eta.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 7
    def __init__(self, timeout: float, custom_timeout: Callable[[object], float]=None):
                              ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.629927 #1]  INFO -- : Skipping file ./tinycrawler/managers/tinycrawler_manager.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 23
    def Statistics(self)->Statistics:
                        ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.643327 #1]  INFO -- : Skipping file ./tinycrawler/log/log.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 6
    def __init__(self, log_filename: str):
                                   ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.665464 #1]  INFO -- : Skipping file ./tinycrawler/utils/utils.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 4
    def get_domain(url: str)->str:
                      ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.695461 #1]  INFO -- : Skipping file ./tinycrawler/statistics/statistics.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 11
    def add(self, category: str, name: str, value: int=1):
                          ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.709183 #1]  INFO -- : Skipping file ./tinycrawler/process/parser.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 16
    def __init__(self, process_spawn_event: Event, process_callback_event: Event, pages_number: Value, urls_number: Value, responses: Queue, urls: Urls, robots: Robots, file_parser: Callable[[str, BeautifulSoup, Log], None], url_validator: Callable[[str, Log], bool], statistics: Statistics, logger: Log, follow_robots_txt: bool, parser_library: str):
                                          ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.729127 #1]  INFO -- : Skipping file ./tinycrawler/process/process_handler.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 11
    def __init__(self, name: str, statistics: Statistics, process_spawn_event: Event):
                           ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.731134 #1]  INFO -- : Skipping file ./tinycrawler/process/downloader.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 18
    def __init__(self, process_spawn_event: Event, process_callback_event: Event, pages_number: Value, urls_number: Value, urls: Urls, local: Local, proxies: Queue, responses: Queue, statistics: Statistics, connection_timeout: float, custom_connection_timeout: Callable[[str], float], maximal_failure_proxy_rate: float, download_attempts: int, cooldown_time_beetween_download_attempts: float):
                                          ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.743771 #1]  INFO -- : Skipping file ./tinycrawler/robots/robots.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 12
    def __init__(self, robots_timeout: float):
                                     ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.759920 #1]  INFO -- : Skipping file ./tinycrawler/proxy/local.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 7
    def __init__(self, domains_timeout: float, custom_domains_timeout: Callable[[str], float], follow_robots_txt: bool, robots: Robots):
                                      ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.767708 #1]  INFO -- : Skipping file ./tinycrawler/proxy/proxy.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 10
    def __init__(self, data: Dict, proxy_timeout: float, domains_timeout: float, custom_domains_timeout: Callable[[str], float], follow_robots_txt: bool, robots: Robots):
                           ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.779161 #1]  INFO -- : Skipping file ./tinycrawler/domains_eta/domains_eta.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 9
    def __init__(self, timeout: float, custom_timeout: Callable[[str], float], follow_robots_txt: bool, robots: Robots):
                              ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.810807 #1]  INFO -- : Skipping file ./tinycrawler/urls/urls.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 11
    def __init__(self, statistics: Statistics, bloom_filters_capacity: int):
                                 ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.816092 #1]  INFO -- : Skipping file ./tinycrawler/cli/cli.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 12
    def __init__(self, statistics, logger, close_signal: Event):
                                                       ^
SyntaxError: invalid syntax


I, [2024-04-08T07:45:13.827961 #1]  INFO -- : Skipping file ./tinycrawler/tinycrawler.py due to exception (CC::Engine::Analyzers::ParserError): `python2 /usr/src/app/lib/cc/engine/analyzers/python/parser.py` exited with code 1:
Traceback (most recent call last):
  File "/usr/src/app/lib/cc/engine/analyzers/python/parser.py", line 51, in <module>
    print(json.dumps(to_json(ast.parse(source))))
  File "/usr/local/python2/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 20
    self, file_parser: Callable[[str, BeautifulSoup, Set[str], Log], None], url_validator: Callable[[str, Log], bool],
                     ^
SyntaxError: invalid syntax
7
apexmetrics
apexmetrics engine documentation
8
bandit
bandit engine documentation
View output
[main]	INFO	profile include tests: None
[main]	INFO	profile exclude tests: None
[main]	INFO	cli include tests: None
[main]	INFO	cli exclude tests: None
[node_visitor]	INFO	Unable to find qualified name for module: setup.py
9
pep8
pep8 engine documentation
View output
tinycrawler/ test_data/ setup.py
10
radon
radon engine documentation
View output
Running radon3...
11
sonar-python
sonar-python engine documentation
View output
INFO: Java 1.8.0_111-internal Oracle Corporation (64-bit)
INFO: Linux 4.4.0-1128-aws amd64
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/src/app/build/libs/sonarlint-core-2.17.0.899.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/src/app/build/libs/sonarlint-cli-2.1.0.566.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
INFO: Index files
INFO: Invalid character encountered in file '/code/preview.png' at line 1 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
INFO: 43 files indexed
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 0.529s
INFO: Final Memory: 6M/295M
INFO: ------------------------------------------------------------------------
12
vint
vint engine documentation