# bigrquery 1.6.2

* BigQuery error messages containing `{` or `}` are no longer mistaken for cli expressions, so the underlying server message is shown instead of a cli parse failure (#677).
* `bq_dataset_query()`, `bq_project_query()`, `bq_table_copy()`, `bq_table_download()`, `bq_table_load()`, `bq_table_save()`, and `bq_table_upload()` now respect `getOption("bigrquery.quiet")` as the default for `quiet` (@r2evans, #663).
* `bq_perform_query()` and friends gain a `labels` argument that attaches [BigQuery labels](https://docs.cloud.google.com/bigquery/docs/labels-intro) to the resulting job, useful for cost allocation. `dbConnect()` gains a matching `labels` argument that is forwarded to every job run on the connection. Defaults to `getOption("bigrquery.labels")` (@JulianUmbhau, #673).
* `bq_perform_upload()` and friends now default to 22 digits of accuracy, and allow you to change this value with the new `json_digits` argument. They now also upload `POSIXt` objects with 6 digits (i.e. microsecond) precision (#660), and correctly set the `"tzone"` attribute of uploaded `POSIXt` columns to `Sys.timezone()` when the timezone is empty or not defined (@r2evans, #660).
* `dbAppendTable()` now correctly uses the billing project from the connection (@eheinzen, #638).
* `dbCreateTable()` and `dbWriteTable()` now correctly say that `temporary = TRUE` (rather than `FALSE`) is unsupported (#670).
* `dbExecute()` gains `params=` support (@r2evans, #667).
* `dbExecute()` and `dbSendQuery()` error if you accidentally use `parameters`  (@r2evans, #667).
* `tbl()` no longer errors when given a literal SQL query that contains three or more periods (#611).

# bigrquery 1.6.1

* Fix test failure on CRAN.

# bigrquery 1.6.0

## New features

* If the bigrquerystorage package is installed, `bq_table_download()` (and
  hence `collect()`, `dbGetQuery()` and `dbFetch()`) will use it. This will
  drastically improve the speed of downloading large datasets. A big thanks
  to @meztez for creating the bigrquerystorage package!

## Bug fixes and minor improvements

* Various R CMD check fixes

* Functions and arguments deprecated in bigrquery 1.4.0 (released) have now
  been removed.

* The `bq_perform_upload()` function now allows users to choose the 
  transmission format (JSON or PARQUET) for data sent to BigQuery 
  (@apalacio9502, #608).

* bigrquery now requires R 4.1, inline with our version support principles.

# bigrquery 1.5.1

* Forward compatibility with upcoming dbplyr release (#601).

# bigrquery 1.5.0

## Major changes

* bigrquery is now MIT licensed (#453).

* Deprecated functions (i.e. those not starting with `bq_`) have been
  removed (#551). These have been superseded for a long time and were formally
  deprecated in bigrquery 1.3.0 (2020).

* `bq_table_download()` now returns unknown fields as character vectors.
  This means that BIGNUMERIC (#435) and JSON (#544) data is downloaded into
  R for you to process as you wish.

  It now parses dates using the clock package. This leads to a considerable
  performance improvement (#430) and ensures that dates prior to 1970-01-01 are
  parsed correctly (#285).

## Significant DBI improvements

* bigquery datasets and tables will now appear in the connection pane when
  using `dbConnect` (@meztez, #431).

* `dbAppendTable()` (#539), `dbCreateTable()` (#483), and `dbExecute` (#502)
  are now supported.

* `dbGetQuery()`/`dbSendQuery()` gains support for parameterised queries via
  the `params` argument (@byapparov, #444).

* `dbReadTable()`, `dbWriteTable()`, `dbExistsTable()`, `dbRemoveTable()`,
  and `dbListFields()` now all work with `DBI::Id()` (#537).

## Significant dbplyr improvements

* bigrquery now uses 2nd edition of dbplyr interface (#508) and is
  compatible with dbplyr 2.4.0 (#550).

* Joins now work correctly across bigrquery connections (#433).

* `grepl(pattern, x)` is now correctly translated to
  `REGEXP_CONTAINS(x, pattern)` (#416).

* `median()` gets a translation that works in `summarise()` and a clear
  error if you use it in `mutate()` (#419).

* `tbl()` now works with views (#519), including the views found in the
  `INFORMATION_SCHEMA` schema (#468).

* `tbl(con, sql("..."))` now works robustly once more (#540), fixing the
  "URL using bad/illegal format or missing URL" error.

* `runif(n())` gains a translation so that `slice_sample()` can work
  (@mgirlich, #448).

## Minor improvements and bug fixes

* Google API URLs have been aligned with the Google Cloud Discovery docs. This
  enables support for Private and Restricted Google APIs configurations
  (@husseyd, #541)

* Functions generally try to do a better job of telling you when you've
  supplied the wrong type of input. Additionally, if you supply `SQL()` to
  a query, you no longer get a weird warning (#498).

* If `bq_job_wait()` receives a 503 response, it now waits for 2 seconds and
  tries again (#535).

* `dbFetch()` now respects the `quiet` setting from the connection (#463).

* `dbGetRowCount()` and `dbHasComplete()` now return correct values when you
  try to fetch more rows than actually exist (#501).

* New `dbQuoteLiteral()` method for logicals reverts breaking change introduced
  by DBI 1.1.2 (@meztez, #478).

* `dbWriteTable()` now correct uses the `billing` value set in the
  connection (#486).

# bigrquery 1.4.2

* Sync up with the current release of gargle (1.4.0). Recently gargle
  introduced some changes around OAuth and bigrquery is syncing with up that:

  - `bq_oauth_client()` is a new function to replace the now-deprecated
    `bq_oauth_app()`.
  -  The new `client` argument of `bq_auth_configure()` replaces the
     now-deprecated `client` argument.
  -  The documentation of `bq_auth_configure()` emphasizes that the preferred
     way to "bring your own OAuth client" is by providing the JSON downloaded
     from Google Developers Console.

* `op_table.lazy_select_query()` now returns a string instead of a list, which
  fixes an error seen when printing or using functions like `head()` or
  `dplyr::glimpse()` (@clente, #509).

# bigrquery 1.4.1

* Fix for `R CMD check` in R-devel (#511)

* bigrquery is now compatible with dbplyr 2.2.0 (@mgirlich, #495).

* brio is new in Imports, replacing the use of the Suggested package readr,
  in `bq_table_download()` (@AdeelK93, #462).

# bigrquery 1.4.0

* `bq_table_download()` has been heavily refactored (#412):

  - It should now return the requested results, in full, in most situations.
    However, when there is a "row shortage", it throws an error instead of
    silently returning incomplete results.
  - The `max_results` argument has been deprecated in favor of `n_max`, which
    reflects what we actually do with this number and is consistent with the
    `n_max` argument elsewhere, e.g., `readr::read_csv()`.
  - The default value of `page_size` is no longer fixed and, instead, is
    determined empirically. Users are strongly recommended to let bigrquery
    select `page_size` automatically, unless there's a specific reason to do
    otherwise.

* The `BigQueryResult` object gains a `billing` slot (@meztez, #423).

* `collect.tbl_BigQueryConnection()` honours the `bigint` field found in a connection object created with `DBI::dbConnect()` and passes `bigint` along to `bq_table_download()`. This improves support for 64-bit integers when reading BigQuery tables with dplyr syntax (@zoews, #439, #437).

# bigrquery 1.3.2

* BigQuery `BYTES` and `GEOGRAPHY` column types are now supported via
  the [blob](https://blob.tidyverse.org/) and
  [wk](https://paleolimbot.github.io/wk/) packages, respectively
  (@paleolimbot, #354, #388).

* When used with dbplyr >= 2.0.0, ambiguous variables in joins will get
  suffixes `_x` and `_y` (instead of `.x` and `.y` which don't work with
  BigQuery) (#403).

* `bq_table_download()` works once again with large row counts
  (@gjuggler, #395). Google's API has stopped accepting `startIndex`
  parameters with scientific formatting, which was happening for large
  values (>1e5) by default.

* New `bq_perform_query_dry_run()` to retrieve the estimated cost of
  performing a query (@Ka2wei, #316).

# bigrquery 1.3.1

* Now requires gargle 0.5.0

# bigrquery 1.3.0

* Old functions (not starting with `bq_`) are deprecated (@byapparov, #335)

* When `bq_perform_*()` fails, you now see all errors, not just the first (#355).

* `bq_perform_query()` can now execute parameterised query with parameters
  of `ARRAY` type (@byapparov, #303). Vectors of length > 1 will be
  automatically converted to `ARRAY` type, or use `bq_param_array()` to
  be explicit.

* `bq_perform_upload()` works once again (#361). It seems like the generated
  JSON was always incorrect, but Google's type checking only recently become
  strict enough to detect the problem.

* `dbExecute()` is better supported. It no longer fails with a spurious
  error for DDL queries, and it returns the number of affected rows for
  DML queries (#375).

* `dbSendQuery()` (and hence `dbGetQuery()`) and `collect()` passes on `...`
  to `bq_perform_query()`. `collect()` gains `page_size` and `max_connection`
  arguments that are passed on to `bq_table_download()` (#374).

* `copy_to()` now works with BigQuery (although it doesn't support temporary
  tables so application is somewhat limited) (#337).

* `str_detect()` now correctly translated to `REGEXP_CONTAINS`
  (@jimmyg3g, #369).

* Error messages include hints for common problems (@deflaux, #353).

# bigrquery 1.2.0

## Auth from gargle

bigrquery's auth functionality now comes from the [gargle package](https://gargle.r-lib.org), which provides R infrastructure to work with Google APIs, in general. The same transition is underway in several other packages, such as [googledrive](https://googledrive.tidyverse.org). This will make user interfaces more consistent and makes two new token flows available in bigrquery:

  * Application Default Credentials
  * Service account tokens from the metadata server available to VMs running on GCE

Where to learn more:

  * Help for [`bq_auth()`](https://bigrquery.r-dbi.org/reference/bq_auth.html) *all that most users need*
  * *details for more advanced users*
    - [How gargle gets tokens](https://gargle.r-lib.org/articles/how-gargle-gets-tokens.html)
    - [Non-interactive auth](https://gargle.r-lib.org/articles/non-interactive-auth.html)
    - [How to get your own API credentials](https://gargle.r-lib.org/articles/get-api-credentials.html)

### Changes that a user will notice

Temporary files are now deleted after table download. (@meztez, #343)

OAuth2 tokens are now cached at the user level, by default, instead of in `.httr-oauth` in the current project. The default OAuth app has also changed. This means you will need to re-authorize bigrquery (i.e. get a new token). You may want to delete any vestigial `.httr-oauth` files lying around your bigrquery projects.

The OAuth2 token key-value store now incorporates the associated Google user when indexing, which makes it easier to switch between Google identities.

`bq_user()` is a new function that reveals the email of the user associated with the current token.

If you previously used `set_service_token()` to use a service account token, it still works. But you'll get a deprecation warning. Switch over to `bq_auth(path = "/path/to/your/service-account.json")`. Several other functions are similarly soft-deprecated.

## Dependency changes

R 3.1 is no longer explicitly supported or tested. Our general practice is to support the current release (3.6), devel, and the 4 previous versions of R (3.5, 3.4, 3.3, 3.2).

gargle and rlang are newly Imported.

# bigrquery 1.1.1

* Fix test failure with dbplyr 1.4.0.

* `bq_field()` can now pass `description` parameter which will be applied
  in `bq_table_create()` call (@byapparov, #272).

* `bq_table_patch()` - allows to patch table (@byapparov, #253) with new schema.


# bigrquery 1.1.0

## Improved type support

* `bq_table_download()` and the `DBI::dbConnect` method now has a `bigint`
  argument which governs how BigQuery integer columns are imported into R. As
  before, the default is `bigint = "integer"`. You can set
  `bigint = "integer64"` to import BigQuery integer columns as
  `bit64::integer64` columns in R which allows for values outside the range of
  `integer` (`-2147483647` to `2147483647`) (@rasmusab, #94).

* `bq_table_download()` now treats NUMERIC columns the same was as FLOAT
  columns (@paulsendavidjay, #282).

* `bq_table_upload()` works with POSIXct/POSIXct variables (#251)

## SQL translation

* `as.character()` now translated to `SAFE_CAST(x AS STRING)` (#268).

* `median()` now translates to `APPROX_QUANTILES(x, 2)[SAFE_ORDINAL(2)]` (@valentinumbach, #267).

## Minor bug fixes and improvements

* Jobs now print their ids while running (#252)

* `bq_job()` tracks location so bigrquery now works painlessly with non-US/EU
  locations (#274).

* `bq_perform_upload()` will only autodetect a schema if the table does
  not already exist.

* `bq_table_download()` correctly computes page ranges if both `max_results`
  and `start_index` are supplied (#248)

* Unparseable date times return NA (#285)

# bigrquery 1.0.0

## Improved downloads

The system for downloading data from BigQuery into R has been rewritten from the ground up to give considerable improvements in performance and flexibility.

* The two steps, downloading and parsing, now happen in sequence, rather than
  interleaved. This means that you'll now see two progress bars: one for
  downloading JSON from BigQuery and one for parsing that JSON into a data
  frame.

* Downloads now occur in parallel, using up to 6 simultaneous connections by
  default.

* The parsing code has been rewritten in C++. As well as considerably improving
  performance, this also adds support for nested (record/struct) and repeated
  (array) columns (#145). These columns will yield list-columns in the
  following forms:

    * Repeated values become list-columns containing vectors.
    * Nested values become list-columns containing named lists.
    * Repeated nested values become list-columns containing data frames.

* Results are now returned as tibbles, not data frames, because the base print
  method does not handle list columns well.

I can now download the first million rows of `publicdata.samples.natality` in about a minute. This data frame is about 170 MB in BigQuery and 140 MB in R; a minute to download this much data seems reasonable to me. The bottleneck for loading BigQuery data is now parsing BigQuery's json format. I don't see any obvious way to make this faster as I'm already using the fastest C++ json parser, [RapidJson](http://rapidjson.org). If this is still too slow for you (i.e. you're downloading GBs of data), see `?bq_table_download` for an alternative approach.

## New features

### dplyr

* `dplyr::compute()` now works (@realAkhmed, #52).

* `tbl()` now accepts fully (or partially) qualified table names, like
  "publicdata.samples.shakespeare" or "samples.shakespeare". This makes it
  possible to join tables across datasets (#219).

### DBI

* `dbConnect()` now defaults to standard SQL, rather than legacy SQL. Use
  `use_legacy_sql = TRUE` if you need the previous behaviour (#147).

* `dbConnect()` now allows `dataset` to be omitted; this is natural when you
  want to use tables from multiple datasets.

* `dbWriteTable()` and `dbReadTable()` now accept fully (or partially)
  qualified table names.

* `dbi_driver()` is deprecated; please use `bigquery()` instead.

### Low-level API

The low-level API has been completely overhauled to make it easier to use. The primary motivation was to make bigrquery development more enjoyable for me, but it should also be helpful to you when you need to go outside of the features provided by higher-level DBI and dplyr interfaces. The old API has been soft-deprecated - it will continue to work, but no further development will occur (including bug fixes). It will be formally deprecated in the next version, and then removed in the version after that.

* __Consistent naming scheme__:
  All API functions now have the form `bq_object_verb()`, e.g.
  `bq_table_create()`, or `bq_dataset_delete()`.

* __S3 classes__:
  `bq_table()`, `bq_dataset()`, `bq_job()`, `bq_field()` and `bq_fields()`
  constructor functions create S3 objects corresponding to important BigQuery
  objects (#150). These are paired with `as_` coercion functions and used throughout
  the new API.

* __Easier local testing__:
  New `bq_test_project()` and `bq_test_dataset()` make it easier to run
  bigrquery tests locally. To run the tests yourself, you need to create a
  BigQuery project, and then follow the instructions in `?bq_test_project`.

* __More efficient data transfer__:
  The new API makes extensive use of the `fields` query parameter, ensuring
  that functions only download data that they actually use (#153).

* __Tighter GCS connection__:
  New `bq_table_load()` loads data from a Google Cloud Storage URI, pairing
  with `bq_table_save()` which saves data to a GCS URI (#155).

## Bug fixes and minor improvements

### dplyr

* The dplyr interface can work with literal SQL once more (#218).

* Improved SQL translation for `pmax()`, `pmin()`, `sd()`, `all()`, and `any()`
  (#176, #179, @jarodmeng). And for `paste0()`, `cor()` and `cov()`
  (@edgararuiz).

* If you have the development version of dbplyr installed, `print()`ing
  a BigQuery table will not perform an unneeded query, but will instead
  download directly from the table (#226).

### Low-level

* Request error messages now contain the "reason", which can contain
  useful information for debugging (#209).

* `bq_dataset_query()` and `bq_project_query()` can now supply query parameters
  (#191).

* `bq_table_create()` can now specify `fields` (#204).

* `bq_perform_query()` no longer fails with empty results (@byapparov, #206).

# Version 0.4.1

* Fix SQL translation omissions discovered by dbplyr 1.1.0

# Version 0.4.0

## New features

* dplyr support has been updated to require dplyr 0.7.0 and use dbplyr. This
  means that you can now more naturally work directly with DBI connections.
  dplyr now also uses modern BigQuery SQL which supports a broader set of
  translations. Along the way I've also fixed some SQL generation bugs (#48).

* The DBI driver gets a new name: `bigquery()`.

* New `insert_extract_job()` make it possible to extract data and save in
  google storage (@realAkhmed, #119).

* New `insert_table()` allows you to insert empty tables into a dataset.

* All POST requests (inserts, updates, copies and `query_exec`) now
  take `...`. This allows you to add arbitrary additional data to the
  request body making it possible to use parts of the BigQuery API
  that are otherwise not exposed (#149). `snake_case` argument names are
  automatically converted to `camelCase` so you can stick consistently
  to snake case in your R code.

* Full support for DATE, TIME, and DATETIME types (#128).

## Big fixes and minor improvements

* All bigrquery requests now have a custom user agent that specifies the
  versions of bigrquery and httr that are used (#151).

* `dbConnect()` gains new `use_legacy_sql`, `page_size`, and `quiet` arguments
  that are passed onto `query_exec()`. These allow you to control query options
  at the connection level.

* `insert_upload_job()` now sends data in newline-delimited JSON instead
  of csv (#97). This should be considerably faster and avoids character
  encoding issues (#45). `POSIXlt` columns are now also correctly
  coerced to TIMESTAMPS (#98).

* `insert_query_job()` and `query_exec()` gain new arguments:

    * `quiet = TRUE` will suppress the progress bars if needed.
    * `use_legacy_sql = FALSE` option allows you to opt-out of the
      legacy SQL system (#124, @backlin)

* `list_tables()` (#108) and `list_datasets()` (#141) are now paginated.
  By default they retrieve 50 items per page, and will iterate until they
  get everything.

* `list_tabledata()` and `query_exec()` now give a nicer progress bar,
  including estimated time remaining (#100).

* `query_exec()` should be considerably faster because profiling revealed that
  ~40% of the time taken by was a single line inside a function that helps
  parse BigQuery's json into an R data frame. I replaced the slow R code with
  a faster C function.

* `set_oauth2.0_cred()` allows user to supply their own Google OAuth
  application when setting credentials (#130, @jarodmeng)

* `wait_for()` uses now reports the query total bytes billed, which is
  more accurate because it takes into account caching and other factors.

* `list_tabledata` returns empty table on max_pages=0 (#184, @ras44 @byapparov)

# Version 0.3.0

* New `set_service_token()` allows you to use OAuth service token instead of
  interactive authentication.from

* `^` is correctly translated to `pow()` (#110).

* Provide full DBI compliant interface (@krlmlr).

* Backend now translates `iflese()` to `IF` (@realAkhmed, #53).

# Version 0.2.0.

* Compatible with latest httr.

* Computation of the SQL data type that corresponds to a given R object
  is now more robust against unknown classes. (#95, @krlmlr)

* A data frame with full schema information is returned for zero-row results.
  (#88, @krlmlr)

* New `exists_table()`. (#91, @krlmlr)

* New arguments `create_disposition` and `write_disposition` to
  `insert_upload_job()`. (#92, @krlmlr)

* Renamed option `bigquery.quiet` to `bigrquery.quiet`. (#89, @krlmlr)

* New `format_dataset()` and `format_table()`. (#81, @krlmlr)

* New `list_tabledata_iter()` that allows fetching a table in chunks of
  varying size. (#77, #87, @krlmlr)

* Add support for API keys via the `BIGRQUERY_API_KEY` environment variable.
  (#49)
