# Introduction-to-rPandas

## Introduction

rPandas is an R package designed to serve as a translation layer, allowing R users to leverage the power and speed of the Python pandas library without ever leaving their R session.

For R users familiar with dplyr or data.table, the rPandas syntax will feel natural. The package provides a set of R functions (e.g., rp_filter(), rp_select()) that:

Capture R code (like carat > 1).

Translate it into an equivalent Python/pandas command string.

Execute the Python code in the background via the reticulate package.

Return the resulting data.frame back to your R session.

This allows you to write R code while the data processing is handled by Python’s pandas library.

## Setup and Installation

### Python Dependencies

rPandas depends on a working Python installation and the pandas library. The reticulate package handles the R‑to‑Python connection.

### Quick installation (default environment)

You can install pandas into a dedicated R environment by running:

    reticulate::py_install("pandas")

#### Troubleshooting the Python environment

If you encounter issues (e.g., “pandas not found”), rPandas provides a built‑in health check:

    # Run this if you have connection issues
    rp_check_env()

The health check prints the Python path reticulate is using and whether `pandas` is installed.

##### Finding and selecting the correct Python environment

You can list all available conda environments (if you use Conda) with:

    reticulate::conda_list()

Once you identify the desired environment (e.g., the first one), you can tell reticulate to use it before loading rPandas:

    # Replace with the path from conda_list()
    reticulate::use_python(python = reticulate::conda_list()$python[1], required = TRUE)

    # Or, if you prefer to use a conda environment by name:
    reticulate::use_condaenv("your_environment_name", required = TRUE)

For system Python installations, simply provide the path to the Python binary:

    reticulate::use_python("/usr/local/bin/python3", required = TRUE)

After setting the environment, rerun `rp_check_env()` to confirm everything is ready. For more detailed guidance, see the reticulate documentation.

> Note: The code chunks above are not executed when building the vignette (they are meant to be run interactively by the user).

## The Core Verbs

All examples will use the diamonds dataset from the ggplot2 package.

    # Make sure ggplot2 is installed to access the data
    data(diamonds, package = "ggplot2")
    head(diamonds)
    #>   carat       cut color clarity depth table price    x    y    z
    #> 1  0.23     Ideal     E     SI2  61.5    55   326 3.95 3.98 2.43
    #> 2  0.21   Premium     E     SI1  59.8    61   326 3.89 3.84 2.31
    #> 3  0.23      Good     E     VS1  56.9    65   327 4.05 4.07 2.31
    #> 4  0.29   Premium     I     VS2  62.4    58   334 4.20 4.23 2.63
    #> 5  0.31      Good     J     SI2  63.3    58   335 4.34 4.35 2.75
    #> 6  0.24 Very Good     J    VVS2  62.8    57   336 3.94 3.96 2.48

1. rp_filter(): This verb filters rows based on a logical expression, similar to dplyr::filter(). It supports common logical operators: & (AND), | (OR), ! (NOT), %in%, and %notin%

    # Simple condition
    v1 <- rp_filter(diamonds, carat > 1)
    print(head(v1))
    #>   carat       cut color clarity depth table price    x    y    z
    #> 1  1.17 Very Good     J      I1  60.2    61  2774 6.83 6.90 4.13
    #> 2  1.01   Premium     F      I1  61.8    60  2781 6.39 6.36 3.94
    #> 3  1.01      Fair     E      I1  64.5    58  2788 6.29 6.21 4.03
    #> 4  1.01   Premium     H     SI2  62.7    59  2788 6.31 6.22 3.93
    #> 5  1.05 Very Good     J     SI2  63.2    56  2789 6.49 6.45 4.09
    #> 6  1.05      Fair     J     SI2  65.8    59  2789 6.41 6.27 4.18

    # AND: multiple conditions
    v2 <- rp_filter(diamonds, carat > 1 & cut == "Ideal")
    print(head(v2))
    #>   carat   cut color clarity depth table price    x    y    z
    #> 1  1.01 Ideal     I      I1  61.5    57  2844 6.45 6.46 3.97
    #> 2  1.02 Ideal     H     SI2  61.6    55  2856 6.49 6.43 3.98
    #> 3  1.02 Ideal     I      I1  61.7    56  2872 6.44 6.49 3.99
    #> 4  1.02 Ideal     J     SI2  60.3    54  2879 6.53 6.50 3.93
    #> 5  1.01 Ideal     I      I1  61.5    57  2896 6.46 6.45 3.97
    #> 6  1.02 Ideal     I      I1  61.7    56  2925 6.49 6.44 3.99

    # OR: use | (pipe)
    v3 <- rp_filter(diamonds, color == "D" | color == "E")
    print(head(v3))
    #>   carat     cut color clarity depth table price    x    y    z
    #> 1  0.23   Ideal     E     SI2  61.5    55   326 3.95 3.98 2.43
    #> 2  0.21 Premium     E     SI1  59.8    61   326 3.89 3.84 2.31
    #> 3  0.23    Good     E     VS1  56.9    65   327 4.05 4.07 2.31
    #> 4  0.22    Fair     E     VS2  65.1    61   337 3.87 3.78 2.49
    #> 5  0.20 Premium     E     SI2  60.2    62   345 3.79 3.75 2.27
    #> 6  0.32 Premium     E      I1  60.9    58   345 4.38 4.42 2.68

    # NOT: use !
    v4 <- rp_filter(diamonds, !(price > 10000))
    print(head(v4))
    #>   carat       cut color clarity depth table price    x    y    z
    #> 1  0.23     Ideal     E     SI2  61.5    55   326 3.95 3.98 2.43
    #> 2  0.21   Premium     E     SI1  59.8    61   326 3.89 3.84 2.31
    #> 3  0.23      Good     E     VS1  56.9    65   327 4.05 4.07 2.31
    #> 4  0.29   Premium     I     VS2  62.4    58   334 4.20 4.23 2.63
    #> 5  0.31      Good     J     SI2  63.3    58   335 4.34 4.35 2.75
    #> 6  0.24 Very Good     J    VVS2  62.8    57   336 3.94 3.96 2.48

    # %in% operator
    v5 <- rp_filter(diamonds, color %in% c("D", "E", "F"))
    print(head(v5))
    #>   carat     cut color clarity depth table price    x    y    z
    #> 1  0.23   Ideal     E     SI2  61.5    55   326 3.95 3.98 2.43
    #> 2  0.21 Premium     E     SI1  59.8    61   326 3.89 3.84 2.31
    #> 3  0.23    Good     E     VS1  56.9    65   327 4.05 4.07 2.31
    #> 4  0.22    Fair     E     VS2  65.1    61   337 3.87 3.78 2.49
    #> 5  0.22 Premium     F     SI1  60.4    61   342 3.88 3.84 2.33
    #> 6  0.20 Premium     E     SI2  60.2    62   345 3.79 3.75 2.27

    # %notin% (if implemented)
    v6 <- rp_filter(diamonds, color %notin% c("D", "E", "F"))
    print(head(v6))
    #>   carat       cut color clarity depth table price    x    y    z
    #> 1  0.29   Premium     I     VS2  62.4    58   334 4.20 4.23 2.63
    #> 2  0.31      Good     J     SI2  63.3    58   335 4.34 4.35 2.75
    #> 3  0.24 Very Good     J    VVS2  62.8    57   336 3.94 3.96 2.48
    #> 4  0.24 Very Good     I    VVS1  62.3    57   336 3.95 3.98 2.47
    #> 5  0.26 Very Good     H     SI1  61.9    55   337 4.07 4.11 2.53
    #> 6  0.23 Very Good     H     VS1  59.4    61   338 4.00 4.05 2.39

2. rp_select(): This verb selects specific columns by name, similar to dplyr::select().

    # Select three columns
    v4 <- rp_select(diamonds, carat, cut, price)
    print(head(v4))
    #>   carat       cut price
    #> 1  0.23     Ideal   326
    #> 2  0.21   Premium   326
    #> 3  0.23      Good   327
    #> 4  0.29   Premium   334
    #> 5  0.31      Good   335
    #> 6  0.24 Very Good   336

3. rp_sort(): This verb sorts the data frame by one or more columns, similar to dplyr::arrange().

    # Sort by price (ascending by default)
    v8 <- rp_sort(diamonds, price)
    print(head(v8))
    #>   carat       cut color clarity depth table price    x    y    z
    #> 1  0.23     Ideal     E     SI2  61.5    55   326 3.95 3.98 2.43
    #> 2  0.21   Premium     E     SI1  59.8    61   326 3.89 3.84 2.31
    #> 3  0.23      Good     E     VS1  56.9    65   327 4.05 4.07 2.31
    #> 4  0.29   Premium     I     VS2  62.4    58   334 4.20 4.23 2.63
    #> 5  0.31      Good     J     SI2  63.3    58   335 4.34 4.35 2.75
    #> 6  0.24 Very Good     I    VVS1  62.3    57   336 3.95 3.98 2.47

    # Use desc() to sort in descending order
    v9 <- rp_sort(diamonds, desc(price))
    print(head(v9))
    #>   carat       cut color clarity depth table price    x    y    z
    #> 1  2.29   Premium     I     VS2  60.8    60 18823 8.50 8.47 5.16
    #> 2  2.00 Very Good     G     SI1  63.5    56 18818 7.90 7.97 5.04
    #> 3  1.51     Ideal     G      IF  61.7    55 18806 7.37 7.41 4.56
    #> 4  2.07     Ideal     G     SI2  62.5    55 18804 8.20 8.13 5.11
    #> 5  2.00 Very Good     H     SI1  62.8    57 18803 7.95 8.00 5.01
    #> 6  2.29   Premium     I     SI1  61.8    59 18797 8.52 8.45 5.24

    # Sort by multiple columns
    v10 <- rp_sort(diamonds, cut, desc(price))
    print(head(v10))
    #>   carat  cut color clarity depth table price     x     y    z
    #> 1  2.01 Fair     G     SI1  70.6    64 18574  7.43  6.64 4.69
    #> 2  2.02 Fair     H     VS2  64.5    57 18565  8.00  7.95 5.14
    #> 3  4.50 Fair     J      I1  65.8    58 18531 10.23 10.16 6.72
    #> 4  2.00 Fair     G     VS2  67.6    58 18515  7.65  7.61 5.16
    #> 5  2.51 Fair     H     SI2  64.7    57 18308  8.44  8.50 5.48
    #> 6  3.01 Fair     I     SI2  65.8    56 18242  8.99  8.94 5.90

4. rp_mutate(): This verb creates new columns or modifies existing ones, similar to dplyr::mutate(). You can also remove columns using the to_remove argument.

    # Create a new column
    v11 <- rp_mutate(diamonds, price_per_carat = price / carat)
    print(head(v11))
    #>   carat       cut color clarity depth table price    x    y    z
    #> 1  0.23     Ideal     E     SI2  61.5    55   326 3.95 3.98 2.43
    #> 2  0.21   Premium     E     SI1  59.8    61   326 3.89 3.84 2.31
    #> 3  0.23      Good     E     VS1  56.9    65   327 4.05 4.07 2.31
    #> 4  0.29   Premium     I     VS2  62.4    58   334 4.20 4.23 2.63
    #> 5  0.31      Good     J     SI2  63.3    58   335 4.34 4.35 2.75
    #> 6  0.24 Very Good     J    VVS2  62.8    57   336 3.94 3.96 2.48
    #>   price_per_carat
    #> 1        1417.391
    #> 2        1552.381
    #> 3        1421.739
    #> 4        1151.724
    #> 5        1080.645
    #> 6        1400.000

    # Create multiple columns
    v12 <- rp_mutate(
      diamonds,
      price_per_carat = price / carat,
      depth_pct = depth / 100
    )
    print(head(v12))
    #>   carat       cut color clarity depth table price    x    y    z
    #> 1  0.23     Ideal     E     SI2  61.5    55   326 3.95 3.98 2.43
    #> 2  0.21   Premium     E     SI1  59.8    61   326 3.89 3.84 2.31
    #> 3  0.23      Good     E     VS1  56.9    65   327 4.05 4.07 2.31
    #> 4  0.29   Premium     I     VS2  62.4    58   334 4.20 4.23 2.63
    #> 5  0.31      Good     J     SI2  63.3    58   335 4.34 4.35 2.75
    #> 6  0.24 Very Good     J    VVS2  62.8    57   336 3.94 3.96 2.48
    #>   price_per_carat depth_pct
    #> 1        1417.391     0.615
    #> 2        1552.381     0.598
    #> 3        1421.739     0.569
    #> 4        1151.724     0.624
    #> 5        1080.645     0.633
    #> 6        1400.000     0.628

    # Remove one or more columns
    v13 <- rp_mutate(diamonds, to_remove = c("table", "depth"))
    print(head(v13))
    #>   carat       cut color clarity price    x    y    z
    #> 1  0.23     Ideal     E     SI2   326 3.95 3.98 2.43
    #> 2  0.21   Premium     E     SI1   326 3.89 3.84 2.31
    #> 3  0.23      Good     E     VS1   327 4.05 4.07 2.31
    #> 4  0.29   Premium     I     VS2   334 4.20 4.23 2.63
    #> 5  0.31      Good     J     SI2   335 4.34 4.35 2.75
    #> 6  0.24 Very Good     J    VVS2   336 3.94 3.96 2.48

5. rp_summarize(): This verb collapses a data frame into a summary, often after grouping. It is similar to dplyr::summarise().

    # Summarize the entire data frame
    v14 <- rp_summarize(diamonds, avg_price = mean(price), max_carat = max(carat))
    print(v14)
    #>    price carat
    #> 1 3932.8   NaN
    #> 2    NaN  5.01

    # Group by one column (unquoted)
    v15 <- rp_summarize(diamonds, avg_price = mean(price), .by = cut)
    print(v15)
    #>         cut avg_price
    #> 1      Fair  4358.758
    #> 2      Good  3928.864
    #> 3 Very Good  3981.760
    #> 4   Premium  4584.258
    #> 5     Ideal  3457.542

    # Group by multiple columns (unquoted)
    v16 <- rp_summarize(
      diamonds,
      avg_price = mean(price),
      count = n(),
      .by = c(cut, color)
    )
    print(head(v16))
    #>    cut color avg_price count
    #> 1 Fair     D  4291.061   163
    #> 2 Fair     E  3682.312   224
    #> 3 Fair     F  3827.003   312
    #> 4 Fair     G  4239.255   314
    #> 5 Fair     H  5135.683   303
    #> 6 Fair     I  4685.446   175

    # Grouping also accepts quoted column names
    v17 <- rp_summarize(
      diamonds,
      avg_price = mean(price),
      .by = c("cut", "color")
    )
    print(head(v17))
    #>    cut color avg_price
    #> 1 Fair     D  4291.061
    #> 2 Fair     E  3682.312
    #> 3 Fair     F  3827.003
    #> 4 Fair     G  4239.255
    #> 5 Fair     H  5135.683
    #> 6 Fair     I  4685.446

6. rp_calculate()– Apply Multiple Functions to Multiple Columns This verb is a powerful alternative to rp_summarize. It applies the same set of functions (e.g., “mean”, “sd”) to every selected column.

    # Apply two functions to two columns, grouped by 'cut'
    v13 <- rp_calculate(
      diamonds,
      price, carat,
      the.functions = c("mean", "sd"),
      .by = cut
    )
    print(head(v13))
    #>         cut price.mean price.std carat.mean carat.std
    #> 1      Fair   4358.758  3560.387  1.0461366 0.5164043
    #> 2      Good   3928.864  3681.590  0.8491847 0.4540544
    #> 3 Very Good   3981.760  3935.862  0.8063814 0.4594354
    #> 4   Premium   4584.258  4349.205  0.8919549 0.5152616
    #> 5     Ideal   3457.542  3808.401  0.7028370 0.4328763

7. rp_first_k_rows() and rp_last_k_rows() – First/Last K Rows per Group These verbs extract the first or last k rows from the whole data frame, or from each group when .by is provided.

    # First 3 rows overall
    v19 <- rp_first_k_rows(diamonds, k = 3)
    print(v19)
    #>   carat     cut color clarity depth table price    x    y    z
    #> 1  0.23   Ideal     E     SI2  61.5    55   326 3.95 3.98 2.43
    #> 2  0.21 Premium     E     SI1  59.8    61   326 3.89 3.84 2.31
    #> 3  0.23    Good     E     VS1  56.9    65   327 4.05 4.07 2.31

    # Last 2 rows per group (cut and clarity)
    v20 <- rp_last_k_rows(diamonds, k = 2, .by = c(cut, clarity))
    print(head(v20))
    #>   carat     cut color clarity depth table price    x    y    z
    #> 1  0.70    Fair     J    VVS1  67.6    54  1691 5.56 5.41 3.71
    #> 2  0.50    Fair     D    VVS1  65.9    64  1792 4.92 5.03 3.28
    #> 3  0.52    Fair     F      IF  64.6    58  2144 5.04 5.17 3.30
    #> 4  0.47    Fair     D      IF  60.6    60  2211 5.09 4.98 3.05
    #> 5  0.55    Good     F      IF  60.8    60  2266 5.26 5.36 3.23
    #> 6  0.54 Premium     F      IF  61.9    60  2391 5.26 5.21 3.24

    # Both quoted and unquoted group specifications work
    v21 <- rp_first_k_rows(diamonds, k = 1, .by = c("cut", "color"))
    print(v21)
    #>    carat       cut color clarity depth table price    x    y    z
    #> 1   0.23     Ideal     E     SI2  61.5  55.0   326 3.95 3.98 2.43
    #> 2   0.21   Premium     E     SI1  59.8  61.0   326 3.89 3.84 2.31
    #> 3   0.23      Good     E     VS1  56.9  65.0   327 4.05 4.07 2.31
    #> 4   0.29   Premium     I     VS2  62.4  58.0   334 4.20 4.23 2.63
    #> 5   0.31      Good     J     SI2  63.3  58.0   335 4.34 4.35 2.75
    #> 6   0.24 Very Good     J    VVS2  62.8  57.0   336 3.94 3.96 2.48
    #> 7   0.24 Very Good     I    VVS1  62.3  57.0   336 3.95 3.98 2.47
    #> 8   0.26 Very Good     H     SI1  61.9  55.0   337 4.07 4.11 2.53
    #> 9   0.22      Fair     E     VS2  65.1  61.0   337 3.87 3.78 2.49
    #> 10  0.23     Ideal     J     VS1  62.8  56.0   340 3.93 3.90 2.46
    #> 11  0.22   Premium     F     SI1  60.4  61.0   342 3.88 3.84 2.33
    #> 12  0.30     Ideal     I     SI2  62.0  54.0   348 4.31 4.34 2.68
    #> 13  0.30      Good     I     SI2  63.3  56.0   351 4.26 4.30 2.71
    #> 14  0.23 Very Good     E     VS2  63.8  55.0   352 3.85 3.92 2.48
    #> 15  0.23 Very Good     G    VVS2  60.4  58.0   354 3.97 4.01 2.41
    #> 16  0.23 Very Good     D     VS2  60.5  61.0   357 3.96 3.97 2.40
    #> 17  0.23 Very Good     F     VS1  60.9  57.0   357 3.96 3.99 2.42
    #> 18  0.23      Good     F     VS1  58.2  59.0   402 4.06 4.08 2.37
    #> 19  0.31      Good     H     SI1  64.0  54.0   402 4.29 4.31 2.75
    #> 20  0.26      Good     D     VS2  65.2  56.0   403 3.99 4.02 2.61
    #> 21  0.23     Ideal     G     VS1  61.9  54.0   404 3.93 3.95 2.44
    #> 22  0.22   Premium     D     VS2  59.3  62.0   404 3.91 3.88 2.31
    #> 23  0.30   Premium     J     SI2  59.3  61.0   405 4.43 4.38 2.61
    #> 24  0.30     Ideal     D     SI1  62.5  57.0   552 4.29 4.32 2.69
    #> 25  0.31   Premium     G     SI1  61.8  58.0   553 4.35 4.32 2.68
    #> 26  0.30   Premium     H     SI1  62.9  59.0   554 4.28 4.24 2.68
    #> 27  0.96      Fair     F     SI2  66.3  62.0  2759 6.27 5.95 4.07
    #> 28  0.81     Ideal     F     SI2  58.8  57.0  2761 6.14 6.11 3.60
    #> 29  0.91      Fair     H     SI2  64.4  57.0  2763 6.11 6.09 3.93
    #> 30  0.77     Ideal     H     VS2  62.0  56.0  2763 5.89 5.86 3.64
    #> 31  0.72      Good     G     VS2  59.7  60.5  2776 5.80 5.84 3.47
    #> 32  0.84      Fair     G     SI1  55.1  67.0  2782 6.39 6.20 3.47
    #> 33  1.05      Fair     J     SI2  65.8  59.0  2789 6.41 6.27 4.18
    #> 34  0.90      Fair     I     SI1  67.3  59.0  2804 5.93 5.84 3.96
    #> 35  0.75      Fair     D     SI2  64.6  57.0  2848 5.74 5.72 3.70

8. rp_count() – Count Rows (Overall or by Group) This verb returns the number of rows in the data frame, optionally by groups.

    # Total row count
    v22 <- rp_count(diamonds)
    print(v22)
    #>       n
    #> 1 53940

    # Count per group
    v23 <- rp_count(diamonds, .by = cut)
    print(v23)
    #>         cut     n
    #> 1      Fair  1610
    #> 2      Good  4906
    #> 3 Very Good 12082
    #> 4   Premium 13791
    #> 5     Ideal 21551

    # Count per combination of multiple groups
    v24 <- rp_count(diamonds, .by = c(cut, color))
    print(head(v24))
    #>    cut color   n
    #> 1 Fair     D 163
    #> 2 Fair     E 224
    #> 3 Fair     F 312
    #> 4 Fair     G 314
    #> 5 Fair     H 303
    #> 6 Fair     I 175

## Chaining Verbs with the Pipe

All rPandas functions are “pipe-friendly” and use .data as their first argument, allowing you to chain operations together using magrittr’s %>% pipe (or the native R |>).

    # Load the pipe
    v25 <- diamonds |>
      rp_filter(carat > 1 & color == "D") |>
      rp_mutate(price_per_carat = price / carat) |>
      rp_summarize(avg_ppc = mean(price_per_carat), .by = cut) |>
      rp_sort(desc(avg_ppc))

    print(head(v25))
    #>         cut  avg_ppc
    #> 1     Ideal 7546.163
    #> 2 Very Good 6789.316
    #> 3   Premium 6548.397
    #> 4      Good 5784.918
    #> 5      Fair 5414.87

## Viewing the Python Code

A key feature of rPandas is its role as a learning and translation tool. Every user-facing verb has a return.as argument that lets you inspect the Python code it generates.

You can set return.as to:

“result” (default): Returns the final R data frame.

“code”: Returns the generated Python command as a string.

“all”: Returns a list containing both the result and the code.

    # See the code for a simple filter
    rp_filter(diamonds, carat > 1 & price < 400, return.as = "code")
    #> [1] "df.query('(carat > 1) and (price < 400)')"

    # See the code for a mutate
    rp_mutate(diamonds, ppc = price / carat, return.as = "code")
    #> [1] "df.assign(ppc = lambda x: (x['price'] / x['carat']))"

    # See the code for a complex summary
    rp_summarize(
      diamonds,
      avg_price = mean(price),
      count = n(),
      .by = c(cut, color),
      return.as = "code"
    )
    #> [1] "df.groupby(['cut', 'color'], as_index=False, observed=True).agg(avg_price = ('price', 'mean'), count = ('price', 'size'))"

## Customizing the Generated Code with table_name

By default, when you request return.as = “code”, the generated Python code uses the placeholder rpandas_df_in as the DataFrame name. This is because the function only receives the data object, not its name. However, you can provide a custom name using the table_name argument (available in all verbs). This is especially useful when you want to copy‑paste the code into a Python script or notebook.

    # Default placeholder
    rp_filter(diamonds, carat > 1, return.as = "code")
    #> [1] "df.query('carat > 1')"

    # With custom table name
    rp_filter(diamonds, carat > 1, table_name = "diamonds", return.as = "code")
    #> [1] "diamonds.query('carat > 1')"

The output changes from: “rpandas_df_in.query(‘(carat > 1)’)” to: “diamonds.query(‘(carat > 1)’)”. This works for all verbs – rp_select, rp_mutate, rp_summarize, etc. Simply pass table_name = “your_data_frame_name” as an argument.