Spaces:
Runtime error
Runtime error
File size: 6,713 Bytes
4a51346 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 |
# fd-slicer
[](https://travis-ci.org/andrewrk/node-fd-slicer)
Safe `fs.ReadStream` and `fs.WriteStream` using the same fd.
Let's say that you want to perform a parallel upload of a file to a remote
server. To do this, we want to create multiple read streams. The first thing
you might think of is to use the `{start: 0, end: 0}` API of
`fs.createReadStream`. This gives you two choices:
0. Use the same file descriptor for all `fs.ReadStream` objects.
0. Open the file multiple times, resulting in a separate file descriptor
for each read stream.
Neither of these are acceptable options. The first one is a severe bug,
because the API docs for `fs.write` state:
> Note that it is unsafe to use `fs.write` multiple times on the same file
> without waiting for the callback. For this scenario, `fs.createWriteStream`
> is strongly recommended.
`fs.createWriteStream` will solve the problem if you only create one of them
for the file descriptor, but it will exhibit this unsafety if you create
multiple write streams per file descriptor.
The second option suffers from a race condition. For each additional time the
file is opened after the first, it is possible that the file is modified. So
in our parallel uploading example, we might upload a corrupt file that never
existed on the client's computer.
This module solves this problem by providing `createReadStream` and
`createWriteStream` that operate on a shared file descriptor and provides
the convenient stream API while still allowing slicing and dicing.
This module also gives you some additional power that the builtin
`fs.createWriteStream` do not give you. These features are:
* Emitting a 'progress' event on write.
* Ability to set a maximum size and emit an error if this size is exceeded.
* Ability to create an `FdSlicer` instance from a `Buffer`. This enables you
to provide API for handling files as well as buffers using the same API.
## Usage
```js
var fdSlicer = require('fd-slicer');
var fs = require('fs');
fs.open("file.txt", 'r', function(err, fd) {
if (err) throw err;
var slicer = fdSlicer.createFromFd(fd);
var firstPart = slicer.createReadStream({start: 0, end: 100});
var secondPart = slicer.createReadStream({start: 100});
var firstOut = fs.createWriteStream("first.txt");
var secondOut = fs.createWriteStream("second.txt");
firstPart.pipe(firstOut);
secondPart.pipe(secondOut);
});
```
You can also create from a buffer:
```js
var fdSlicer = require('fd-slicer');
var slicer = FdSlicer.createFromBuffer(someBuffer);
var firstPart = slicer.createReadStream({start: 0, end: 100});
var secondPart = slicer.createReadStream({start: 100});
var firstOut = fs.createWriteStream("first.txt");
var secondOut = fs.createWriteStream("second.txt");
firstPart.pipe(firstOut);
secondPart.pipe(secondOut);
```
## API Documentation
### fdSlicer.createFromFd(fd, [options])
```js
var fdSlicer = require('fd-slicer');
fs.open("file.txt", 'r', function(err, fd) {
if (err) throw err;
var slicer = fdSlicer.createFromFd(fd);
// ...
});
```
Make sure `fd` is a properly initialized file descriptor. If you want to
use `createReadStream` make sure you open it for reading and if you want
to use `createWriteStream` make sure you open it for writing.
`options` is an optional object which can contain:
* `autoClose` - if set to `true`, the file descriptor will be automatically
closed once the last stream that references it is closed. Defaults to
`false`. `ref()` and `unref()` can be used to increase or decrease the
reference count, respectively.
### fdSlicer.createFromBuffer(buffer, [options])
```js
var fdSlicer = require('fd-slicer');
var slicer = fdSlicer.createFromBuffer(someBuffer);
// ...
```
`options` is an optional object which can contain:
* `maxChunkSize` - A `Number` of bytes. see `createReadStream()`.
If falsey, defaults to unlimited.
#### Properties
##### fd
The file descriptor passed in. `undefined` if created from a buffer.
#### Methods
##### createReadStream(options)
Available `options`:
* `start` - Number. The offset into the file to start reading from. Defaults
to 0.
* `end` - Number. Exclusive upper bound offset into the file to stop reading
from.
* `highWaterMark` - Number. The maximum number of bytes to store in the
internal buffer before ceasing to read from the underlying resource.
Defaults to 16 KB.
* `encoding` - String. If specified, then buffers will be decoded to strings
using the specified encoding. Defaults to `null`.
The ReadableStream that this returns has these additional methods:
* `destroy(err)` - stop streaming. `err` is optional and is the error that
will be emitted in order to cause the streaming to stop. Defaults to
`new Error("stream destroyed")`.
If `maxChunkSize` was specified (see `createFromBuffer()`), the read stream
will provide chunks of at most that size. Normally, the read stream provides
the entire range requested in a single chunk, but this can cause performance
problems in some circumstances.
See [thejoshwolfe/yauzl#87](https://github.com/thejoshwolfe/yauzl/issues/87).
##### createWriteStream(options)
Available `options`:
* `start` - Number. The offset into the file to start writing to. Defaults to
0.
* `end` - Number. Exclusive upper bound offset into the file. If this offset
is reached, the write stream will emit an 'error' event and stop functioning.
In this situation, `err.code === 'ETOOBIG'`. Defaults to `Infinity`.
* `highWaterMark` - Number. Buffer level when `write()` starts returning
false. Defaults to 16KB.
* `decodeStrings` - Boolean. Whether or not to decode strings into Buffers
before passing them to` _write()`. Defaults to `true`.
The WritableStream that this returns has these additional methods:
* `destroy()` - stop streaming
And these additional properties:
* `bytesWritten` - number of bytes written to the stream
And these additional events:
* 'progress' - emitted when `bytesWritten` changes.
##### read(buffer, offset, length, position, callback)
Equivalent to `fs.read`, but with concurrency protection.
`callback` must be defined.
##### write(buffer, offset, length, position, callback)
Equivalent to `fs.write`, but with concurrency protection.
`callback` must be defined.
##### ref()
Increase the `autoClose` reference count by 1.
##### unref()
Decrease the `autoClose` reference count by 1.
#### Events
##### 'error'
Emitted if `fs.close` returns an error when auto closing.
##### 'close'
Emitted when fd-slicer closes the file descriptor due to `autoClose`. Never
emitted if created from a buffer.
|