Comparison of Quantiles of Two Groups with Permutation Tests
Author: Zeynel Cebeci, A. Firat Ozdemir, Engin Yildiztepe
Date: 2025-05-05

# 1 Introduction

This vignette provides a comprehensive guide for performing permutation tests to compare quantiles of two groups in R. A permutation test is a non-parametric statistical method that involves rearranging the data points to generate a distribution of a test statistic under the null hypothesis. This technique is particularly useful when the assumptions required for traditional parametric tests are not met, providing a robust alternative for statistical inference.

In experiments involving two groups, it is essential to assess whether there are significant differences between their distributions. Traditional methods might focus on comparing means or medians, but quantile comparison offers a deeper insight into the distributional characteristics of the groups. Quantiles, such as the median (50th percentile) or quartiles (25th and 75th percentiles), divide the data into equal parts and highlight specific points in the dataset that are critical for understanding the distribution’s shape and spread.

This vignette demonstrates how to implement permutation tests for quantile comparison using the `permtest` function in the package `groupcompare`. By following the steps outlined, users will be able to conduct thorough statistical analyses and draw meaningful conclusions about their data.

# 2 Required Packages and Data Sets

## 2.1 Install and load the package gpcomp

The recent version of the package from CRAN is installed with the following command:

    install.packages("groupcompare", dep=TRUE)

If you have already installed `groupcompare`, you can load it into R working environment by using the following command:

    library("groupcompare")

## 2.2 Data sets

The dataset to be analyzed can be in wide format where the values for Group 1 and Group 2 are written in two separate columns or in long format where the values are entered in the first column and the group names are entered in the second column. In the following code chunk, a dataset named `ds1` is created using the `ghdist` function to simulate the G&H distribution. The generated dataset contains data for two groups named `A` and `B`, each consisting of 25 observations, with a mean of 50 and a standard deviation of 2. In the example, by assigning zeros to the skewness (`g`) and kurtosis (`h`) arguments, the simulated data is intended to have a normal distribution. As expected, the means and variances of groups `A` and `B` are done equal. In the example, the generated dataset is in wide format, and immediately after, it is converted to long format using the wide2long function to create the dataset `ds2`. This provides an idea of the long data format, and as can be seen, in the long data format, the first column contains the observation values, while the second column contains the group names or codes. As understood from the example, different groups can be created by changing the means, variances, skewness, and kurtosis parameters.

     set.seed(12) # For reproducibility purpose
     grp1 <- ghdist(50, 50, 2, g=0, h=0)
     grp2 <- ghdist(50, 45, 4, g=0.8, h=0)
     ds1 <- data.frame(grp1=grp1, grp2=grp2)
     head(ds1)

    ##       grp1     grp2
    ## 1 47.03886 44.83214
    ## 2 53.15434 44.56903
    ## 3 48.08651 47.20590
    ## 4 48.15999 65.17133
    ## 5 46.00472 42.15702
    ## 6 49.45541 48.99942

     # Data in long format
     ds2 <- wide2long(ds1)
     head(ds2)

    ##        obs group
    ## 1 47.03886  grp1
    ## 2 53.15434  grp1
    ## 3 48.08651  grp1
    ## 4 48.15999  grp1
    ## 5 46.00472  grp1
    ## 6 49.45541  grp1

# 3 Data Visualization

For statistical tests, data visualization is performed before the analysis to provide insights about the structure or distribution of the data. In the comparison of two groups using parametric tests such as the t-test, visualization provides preliminary information on whether the assumptions of the test are met. The `bivarplot` function in the following code chunk facilitates the examination and comparison of group data using various plots.

    bivarplot(ds2)

# 4 Resampling Statistics with Permutations

The `permtest` function of the package `groupcompare`, given an example of its usage in the following code chunk, compares the groups in the dataset using permutations and returns the results. In the result object, `pval` contains the p-values of for the differences of each statictic related to two compared groups. (Please note that the value of R should be higher (e.g. 3000) in real-world applications.)

    results <- permtest(ds2, statistic=calcquantdif, alternative="two.sided", R=500)

In the code chunk above, `ds2` is the name of dataset in long data format in which the group names locate in the second column. As a mandatory argument, the `statistic` is the name of the function that calculates and returns the quantile differences of the compared groups. In the example, `calcquantdif` is a function that calculates and returns the differences between group quantiles. Among its arguments, `alternative` shows the type of null hyphothesis. The default is `two.sided` but it can be set to `less` or `greater` for a single-tail alternative hyphothesis as well. `R` shows the number of repetitions for bootstrap. The default number of permutations is 5000 but it is recommended to increase up to a number as high as 20000.

# 5 Results and Interpretation

Below, the code chunk displays the structure of `results` object and the results stored obtained above. Here, `results$tstar` stores the statistics obtained in permutations and can be used for further visualization and analysis. As it is seen, the p-values are `2.20e-16` for the difference between all of the quantiles, meaning all the quantiles are siginificantly different.

    str(results)

    ## List of 7
    ##  $ t0         : Named num [1:9] 5.62 4.95 4.86 4.98 4.5 ...
    ##   ..- attr(*, "names")= chr [1:9] "P10" "P20" "P30" "P40" ...
    ##  $ tstar      : num [1:500, 1:9] -1.325 0.523 -0.974 -0.109 -0.641 ...
    ##   ..- attr(*, "dimnames")=List of 2
    ##   .. ..$ : NULL
    ##   .. ..$ : chr [1:9] "P10" "P20" "P30" "P40" ...
    ##  $ pval       : Named num [1:9] 2.2e-16 2.2e-16 2.2e-16 2.2e-16 2.2e-16 ...
    ##   ..- attr(*, "names")= chr [1:9] "P10" "P20" "P30" "P40" ...
    ##  $ alternative: chr "two.sided"
    ##  $ R          : num 500
    ##  $ pvalsk     : logi [1:2] NA NA
    ##  $ call       : language permtest(x = ds2, statistic = calcquantdif, alternative = "two.sided",      R = 500)

    results$t0

    ##       P10       P20       P30       P40       P50       P60       P70       P80
    ## 5.6160059 4.9501201 4.8594475 4.9845448 4.4981128 3.5381490 2.8770037 0.9401363
    ##       P90
    ## 0.1316084

    head(results$tstar)

    ##             P10        P20        P30        P40         P50          P60
    ## [1,] -1.3245136 -1.7530604 -2.6483186 -1.0072093 -1.23359354 -0.229271264
    ## [2,]  0.5231460 -0.3013630 -0.3761942 -0.6354308 -1.42081451 -0.866892293
    ## [3,] -0.9744594  0.1857390 -0.3761942  0.4736984 -0.01297126  0.259526066
    ## [4,] -0.1085207  0.6999764  1.2457520  0.1484691 -0.53111609  0.013482962
    ## [5,] -0.6413481  0.2435761  1.3410715  0.6504253 -0.06140080  0.009553991
    ## [6,] -0.2829107  0.5266653  1.1272236  0.9468752  0.61473382 -0.139181597
    ##             P70         P80         P90
    ## [1,] -0.5960080 -0.64748245 -1.03102517
    ## [2,] -0.8817546 -0.46472389 -0.78346152
    ## [3,]  0.5389690  0.51634212  0.10362627
    ## [4,]  0.3276260  0.03798994  1.15707616
    ## [5,]  0.4385084  0.14315489 -0.08459354
    ## [6,] -0.4630248 -0.63551890 -1.18610636

    results$pval

    ##      P10      P20      P30      P40      P50      P60      P70      P80
    ## 2.20e-16 2.20e-16 2.20e-16 2.20e-16 2.20e-16 2.20e-16 2.20e-16 6.20e-02
    ##      P90
    ## 9.46e-01

# 6 Other Tests

In addition to permutation test, performing bootstrap can be useful for validation of the results when comparing the quantiles of two groups. For this purpose, the `bootstrap` function in the `groupcompare` package can be run. For details, you can refer to the usage documentation of the package as well as the vignette titled Quantiles Comparison of Two Groups with Bootstrap.

While p-values for the difference of percentiles of two groups have been calculated here, significance differences for other group statistics can be determined by passing different function names to the `statistic` argument. For example, when the `calcstatdif` function is assigned to the `statistic` argument in the example above, p-values can be calculated for the differences between the means, medians, IQRs, and variances of the two groups.