# pxf_regress

## Overview

`pxf_regress` is a PSQL test runner written in Go that is heavily inspired by
`pg_regress`. PXF's automation test framework sets up data in external data
storage (e.g., Hadoop, Amazon S3, etc), creates Cloudberry external tables to
work with these data sets, and then invokes `pxf_regress` to run SQL test cases
via `psql` and compare the results with expected output. Instead of matching
the features of `pg_regress` exactly, this utility currently implements the
minimal set of features required by PXF's usage (e.g., no support for schedule
files, alternate expected files, resultmaps, etc).

### Usage

Normally, `pxf_regress` is orchestrated by the automation test framework and
developers shouldn't need to run the utility directly; developers should be
using `make` in the parent directory to run automation tests. Developers can
run `pxf_regress` from this directory:

```console
usage: pxf_regress <test-directory>
```

Here `<test-directory>` is a path (relative or absolute) to a directory that
contains `expected` and `sql` subdirectories:

```console
$ tree small_data
small_data
├── expected
│   ├── query01.ans
│   └── query02.ans
└── sql
    ├── query01.sql
    └── query02.sql
```

There are no command line flags; the Cloudberry cluster that `pxf_regress` connects
to can be customized with standard [Postgres environment variables][1].

### Why not use `pg_regress`?

Ideally, PXF would re-use `pg_regress` which is included with upstream Cloudberry;
however, PXF supports multiple Cloudberry versions with a
single code base. Differences between the Cloudberry major versions and the included
`pg_regress` results in non-semantically meaningful (for PXF) differences.
Cloudberry's version of `pg_regress` uses a utility called `gpdiff.pl` to compare
actual test output with expected test output. From the description of
[`gpdiff.pl`][2]:

> gpdiff compares files using diff after processing them with atmsort.pm.
> This comparison is designed to ignore certain Greenplum-specific
> informational messages, as well as handle the cases where query output
> order may differ for a multi-segment Greenplum database versus a
> single PostgreSQL instance.

When `pg_regress` runs `gpdiff.pl`, it runs the version of `gpdiff.pl` that is
included with Cloudberry (`$($GPHOME/bin/pg_config
--libdir)/postgresql/pgxs/src/test/regress/gpdiff.pl`) with hard-coded options
that cannot be customized. Not only is `gpdiff.pl` different across Cloudberry major
versions, the set of options that `pg_regress` runs it with will be different
across major versions.

The existing set of PXF test cases (of which there are over 450) assume that
actual and expected test output will be compared in such a way that

1. white space and blank lines will be ignored
1. headers for query results will be ignored

While it may be possible to update the current set of test cases for PXF to be
compatible with `pg_regress` via combination of updating expected output,
alternate expected output files, and additional match/subs, doing so would
require an extensive amount of work for little gain **and** introduce a risk of
regression by introducing subtle changes to the expected output that hides
meaningful differences.

## Debugging Failing Tests

When `pxf_regress` runs, it will write several files that can be used to
identify the error. Shown below is an example directory structure of the files
created by `pxf_regress`

```console
$ tree smoke/small_data
├── regression.diffs
└── output
    ├── query01_<timestamp>.out
    ├── query01_<timestamp>.diff
    ├── query02_<timestamp>.out
    └── query02_<timestamp>.out
```

1. View the contents of
   `automation_logs/<test-class>/<timestamp>_<test-method>.log` for the failing
   test.

    These are log message generated by the Java automation project and will
    help identify if the error is during the test setup phase (i.e., staging
    data in an external system) or during the running of `pxf_regress`. Log
    messages related to running `pxf_regress` will be prefixed with `Regress
    ->`.

1. If the failure occurs while running `pxf_regress`, you can view all of
   differences between the actual results and the expected results in
   `<test-directory>/regression.diffs`; individual diffs can be viewed in
   `<test-directory>/output/<test-name>_<timestamp>.diff`.

1. You can view the output of running the test query with `psql` by viewing
   `<test-directory>/results/<test-name>.out`; here is an example:

    ```console
    $ cat ./smoke/small_data/results/query01.out
    -- @description query01 for PXF test on small data
    SELECT *  FROM pxf_smoke_small_data ORDER BY name;
    psql:smoke/small_data/sql/query01.sql:3: ERROR:  PXF server error : Input path does not exist: hdfs://alpine:8020/tmp/pxf_automation_data/10f83dae-5146-44d7-957d-ea335cb3ea24/data.txt  (seg0 slice1 127.0.1.1:6000 pid=115042)
    HINT:  Check the PXF logs located in the '/home/bradford/pxf-base/logs' directory on host 'localhost' or 'set client_min_messages=LOG' for additional details.
    CONTEXT:  External table pxf_smoke_small_data, line 1 of file pxf://tmp/pxf_automation_data/10f83dae-5146-44d7-957d-ea335cb3ea24/data.txt?PROFILE=hdfs:text
    ```

[1]: https://www.postgresql.org/docs/12/libpq-envars.html
[2]: https://github.com/apache/cloudberry/blob/main/src/test/regress/gpdiff.pl
