Skip to content

fix: Include reason in the session rotation warning logs #1363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 19, 2025
Merged

fix: Include reason in the session rotation warning logs #1363

merged 1 commit into from
Aug 19, 2025

Conversation

vdusek
Copy link
Collaborator

@vdusek vdusek commented Aug 19, 2025 •
edited
Loading

Description

  • Include reason in the session rotation warning logs.

Issues

Reproduction

import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext
from crawlee.errors import SessionError


async def main() -> None:
    crawler = ParselCrawler()

    @crawler.router.default_handler
    async def request_handler(_: ParselCrawlingContext) -> None:
        raise SessionError('Blah blah blah')

    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

Log:

[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 2; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
[ParselCrawler] WARN  Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN  Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN  Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN  Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN  Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN  Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN  Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN  Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] WARN  Encountered "crawlee.errors.SessionError: Blah blah blah", rotating session and retrying...
[ParselCrawler] ERROR Request to https://crawlee.dev failed and reached maximum retries
 Traceback (most recent call last):
  File "/home/vdusek/Projects/crawlee-python/src/crawlee/crawlers/_basic/_basic_crawler.py", line 1312, in __run_task_function
    await self._run_request_handler(context=context)
  File "/home/vdusek/Projects/crawlee-python/src/crawlee/crawlers/_basic/_basic_crawler.py", line 1407, in _run_request_handler
    await wait_for(
    ...<5 lines>...
    )
  File "/home/vdusek/Projects/crawlee-python/src/crawlee/_utils/wait.py", line 37, in wait_for
    return await asyncio.wait_for(operation(), timeout.total_seconds())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vdusek/.local/share/uv/python/cpython-3.13.0-linux-x86_64-gnu/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/vdusek/Projects/crawlee-python/src/crawlee/crawlers/_basic/_context_pipeline.py", line 114, in __call__
    await final_context_consumer(cast('TCrawlingContext', crawling_context))
  File "/home/vdusek/Projects/crawlee-python/src/crawlee/router.py", line 98, in __call__
    return await self._default_handler(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vdusek/Projects/crawlee-python/run_crawler.py", line 12, in request_handler
    raise SessionError('Blah blah blah')
crawlee.errors.SessionError: Blah blah blah
[crawlee._autoscaling.autoscaled_pool] INFO  Waiting for remaining tasks to finish

Checklist

  • CI passed

@vdusek vdusek added this to the 121st sprint - Tooling team milestone Aug 19, 2025
@vdusek vdusek requested a review from Pijukatel August 19, 2025 11:44
@vdusek vdusek self-assigned this Aug 19, 2025
@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Aug 19, 2025
@vdusek vdusek changed the title fix: Include reason to session rotation log warnings fix: Include reason in the session rotation log warnings Aug 19, 2025
@vdusek vdusek changed the title fix: Include reason in the session rotation log warnings fix: Include reason in the session rotation warning logs Aug 19, 2025
@vdusek vdusek merged commit d6d7a45 into master Aug 19, 2025
39 of 41 checks passed
@vdusek vdusek deleted the include-reason-to-session-rotation-warning branch August 19, 2025 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Pijukatel Pijukatel Pijukatel approved these changes

Assignees

@vdusek vdusek

Labels
t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Milestone
121st sprint - Tooling team
Development

Successfully merging this pull request may close these issues.

Include reason to session rotation warnings
2 participants
@vdusek @Pijukatel