Skip to content

Clarify logs processors error handling expectations #4724

@mladjan-gadzic

Description

@mladjan-gadzic

Describe your environment

OS: (e.g, MacOS)
Python version: (e.g., Python 3.9.6)
SDK version: (e.g., 1.33.1)
API version: (e.g., 1.33.1)

What happened?

Unhandled exception occurs when LogExporter.export() is called and LogRecordProcessor did not guard against exceptions, which causes exception to bubble up and crash program. It should be LogExporter job to guard against that not LogRecordProcessor.

Steps to Reproduce

Custom processor:

from opentelemetry.sdk._logs import LogRecordProcessor

class RawLogProcessor(LogRecordProcessor):
    def __init__(self, exporter):
        self.exporter = exporter

    def emit(self, log_data):
        self.exporter.export([log_data])

    def shutdown(self):
        if hasattr(self.exporter, 'shutdown'):
            self.exporter.shutdown()

    def force_flush(self, timeout_millis=30000):
        if hasattr(self.exporter, 'force_flush'):
            return self.exporter.force_flush(timeout_millis)
        return True

Reproduction program:

from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from raw_log_processor import RawLogProcessor
import logging

otlp_exporter = OTLPLogExporter(endpoint="http://10.255.255.1:4318/v1/logs", timeout=1)
logger_provider = LoggerProvider()
logger_provider.add_log_record_processor(RawLogProcessor(otlp_exporter))
set_logger_provider(logger_provider)

logger = logging.getLogger(__name__)
logger.addHandler(LoggingHandler(logger_provider=logger_provider))
logger.setLevel(logging.INFO)
logger.propagate = False

logger.info("test")

print("If you see this, the logging worked!")

Expected Result

According to OTEL principles https://opentelemetry.io/docs/specs/otel/error-handling/#basic-error-handling-principles, SDK should not throw unhandled exceptions. For example, if ConnectTimeout occurs during OTLPLogExporter._export() method, LogExporter should handle it and not throw.

Actual Result

SDK does not handle exception which bubble up and cause program to crash.

Exception: requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='10.255.255.1', port=4318): Max retries exceeded with url: /v1/logs (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x104316a30>, 'Connection to 10.255.255.1 timed out. (connect timeout=1)'))

Additional context

The fix would be to handle exceptions in each LogExporter export method implementation.

Would you like to implement a fix?

Yes

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions