Skip to content

refactor: Unify compile paths with ResultNode #1636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 21, 2025
Merged

Conversation

TrevorBergeron
Copy link
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@product-auto-label product-auto-label bot added size: l Pull request size is large. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels Apr 19, 2025
@TrevorBergeron TrevorBergeron marked this pull request as ready for review April 21, 2025 22:10
@TrevorBergeron TrevorBergeron requested review from a team as code owners April 21, 2025 22:10
@TrevorBergeron TrevorBergeron requested a review from sycai April 21, 2025 22:10
@TrevorBergeron TrevorBergeron changed the title refactor: Simplify compile with ResultNode refactor: Unify compile paths with ResultNode Apr 21, 2025
node: nodes.BigFrameNode
sort_rows: bool
materialize_all_order_keys: bool = False
peek_rows: typing.Optional[int] = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

naming nit: "peek_count" to indicate that is a numeric value?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

distinct: bool = False,
) -> Select:
if isinstance(columns, str):
columns = [columns]
self.select_list: typing.List[typing.Union[SelectExpression, SelectAll]] = (
[
SelectExpression(expression=expr.ColumnExpression(name=column))
if isinstance(column, str)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is probably complicated enough to be defined in a separate helper function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@property
def row_count(self) -> Optional[int]:
child_count = self.child.row_count
if child_count is not None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we unnest the code?

if child_count is None:
  return None
if self.limit is None:
  return child_count
return min(self.limit, child_count)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

output_names: tuple[str, ...]
order_by: Tuple[OrderingExpression, ...] = ()
output_cols: tuple[tuple[ex.DerefOp, str], ...]
is_ordered: bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is is_ordered necessary? Can we rely on the nullness of order_by to decide whether it is ordered?

If not, we might need to document how we are going to handle this confusing case when is_ordered == False and order_by == None

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, not needed, removed

result_node = rewrites.defer_order(
result_node, output_hidden_row_keys=request.materialize_all_order_keys
)
if not request.sort_rows:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style nit: let's re-write this if-else as:

if request.sort_rows:
...
else:
...

(the "else" indentation might be unnecessary too, because the "if" branch always returns)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@TrevorBergeron TrevorBergeron requested a review from sycai April 21, 2025 23:14
@TrevorBergeron TrevorBergeron merged commit 3c314c3 into main Apr 21, 2025
19 of 24 checks passed
@TrevorBergeron TrevorBergeron deleted the output_node2 branch April 21, 2025 23:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants