Write the data produced by the chunks to their output files. This is mostly a wrapper around write_csv that figures out file names, processes table flags, writes metadata comments, and so forth.

save_chunkdata(
  chunkdata,
  write_inputs = FALSE,
  create_dirs = FALSE,
  write_outputs = TRUE,
  write_xml = write_outputs,
  outputs_dir = OUTPUTS_DIR,
  xml_dir = XML_DIR
)

Arguments

chunkdata

Named list of tibbles (data frames) to write

write_inputs

Write data that were read as inputs, not computed? Logical

create_dirs

Create directory if necessary, and delete contents? Logical

write_outputs

Write all chunk outputs to disk?

write_xml

Write XML Batch chunk outputs to disk?

outputs_dir

Directory to save data into

xml_dir

Directory to save XML results into

Details

One thing to be aware of is that there is a wart in readr v 1.1 and later that causes floating point data to be written as integers if they happen to have integer values. This can cause problems if there are so many apparently-integer values before the first obviously-float value that read_csv concludes that the column should have integer type. If this seems to be happening to your table, add the PROTECT_FLOAT flag to it, and any floating point data in your table will be protected before it is written. Use this option sparingly, as the data written that way tends to be a lot bigger, owing to the large number of digits we have to write.