public class WriterImpl extends Object implements WriterInternal, MemoryManager.Callback
This class is unsynchronized like most Stream objects, so from the creation of an OrcFile and all access to a single instance has to be from a single thread.
There are no known cases where these happen between different threads today.
Caveat: the MemoryManager is created during WriterOptions create, that has to be confined to a single thread as well.
| Constructor and Description |
|---|
WriterImpl(FileSystem fs,
Path path,
OrcFile.WriterOptions opts) |
| Modifier and Type | Method and Description |
|---|---|
void |
addRowBatch(VectorizedRowBatch batch)
Add a row batch to the ORC file.
|
void |
addUserMetadata(String name,
ByteBuffer value)
Add arbitrary meta-data to the ORC file.
|
void |
appendStripe(byte[] stripe,
int offset,
int length,
StripeInformation stripeInfo,
OrcProto.StripeStatistics stripeStatistics)
Fast stripe append to ORC file.
|
void |
appendStripe(byte[] stripe,
int offset,
int length,
StripeInformation stripeInfo,
StripeStatistics[] stripeStatistics)
Fast stripe append to ORC file.
|
void |
appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata)
Update the current user metadata with a list of new values.
|
boolean |
checkMemory(double newScale)
The scale factor for the stripe size has changed and thus the
writer should adjust their desired size appropriately.
|
void |
close()
Flush all of the buffers and close the file.
|
static CompressionCodec |
createCodec(CompressionKind kind) |
long |
estimateMemory()
Estimate the memory currently used by the writer to buffer the stripe.
|
CompressionCodec |
getCompressionCodec() |
static int |
getEstimatedBufferSize(long stripeSize,
int numColumns,
int bs) |
long |
getNumberOfRows()
Row count gets updated when flushing the stripes.
|
long |
getRawDataSize()
Raw data size will be compute when writing the file footer.
|
TypeDescription |
getSchema()
Get the schema for this writer
|
ColumnStatistics[] |
getStatistics()
Get the statistics about the columns in the file.
|
List<StripeInformation> |
getStripes()
Get the stripe information about the file.
|
void |
increaseCompressionSize(int newSize)
Increase the buffer size for this writer.
|
long |
writeIntermediateFooter()
Write an intermediate footer on the file such that if the file is
truncated to the returned offset, it would be a valid ORC file.
|
public WriterImpl(FileSystem fs, Path path, OrcFile.WriterOptions opts) throws IOException
IOExceptionpublic static int getEstimatedBufferSize(long stripeSize,
int numColumns,
int bs)
public void increaseCompressionSize(int newSize)
WriterInternalincreaseCompressionSize in interface WriterInternalnewSize - the new buffer size.public static CompressionCodec createCodec(CompressionKind kind)
public boolean checkMemory(double newScale)
throws IOException
MemoryManager.CallbackcheckMemory in interface MemoryManager.CallbacknewScale - the current scale factor for memory allocationsIOExceptionpublic TypeDescription getSchema()
Writerpublic void addUserMetadata(String name, ByteBuffer value)
WriteraddUserMetadata in interface Writername - a key to label the data with.value - the contents of the metadata.public void addRowBatch(VectorizedRowBatch batch) throws IOException
WriteraddRowBatch in interface Writerbatch - the rows to addIOExceptionpublic void close()
throws IOException
Writerclose in interface Closeableclose in interface AutoCloseableclose in interface WriterIOExceptionpublic long getRawDataSize()
getRawDataSize in interface Writerpublic long getNumberOfRows()
getNumberOfRows in interface Writerpublic long writeIntermediateFooter()
throws IOException
WriterwriteIntermediateFooter in interface WriterIOExceptionpublic void appendStripe(byte[] stripe,
int offset,
int length,
StripeInformation stripeInfo,
OrcProto.StripeStatistics stripeStatistics)
throws IOException
WriterWriter.appendStripe(byte[], int, int, StripeInformation, StripeStatistics[])
for files with encryption.appendStripe in interface Writerstripe - - stripe as byte arrayoffset - - offset within byte arraylength - - length of stripe within byte arraystripeInfo - - stripe informationstripeStatistics - - unencrypted stripe statisticsIOExceptionpublic void appendStripe(byte[] stripe,
int offset,
int length,
StripeInformation stripeInfo,
StripeStatistics[] stripeStatistics)
throws IOException
WriterWriter.addUserMetadata(String,
ByteBuffer) to append any user metadata.appendStripe in interface Writerstripe - - stripe as byte arrayoffset - - offset within byte arraylength - - length of stripe within byte arraystripeInfo - - stripe informationstripeStatistics - - stripe statistics with the last one being
for the unencrypted data and the others being for
each encryption variant.IOExceptionpublic void appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata)
WriterappendUserMetadata in interface WriteruserMetadata - - user metadatapublic ColumnStatistics[] getStatistics()
WritergetStatistics in interface Writerpublic List<StripeInformation> getStripes() throws IOException
WritergetStripes in interface WriterIOExceptionpublic CompressionCodec getCompressionCodec()
public long estimateMemory()
WriterestimateMemory in interface WriterCopyright © 2013–2023 The Apache Software Foundation. All rights reserved.