Skip to content

Commit

Permalink
Merge pull request #112 from frictionlessdata/update-doc
Browse files Browse the repository at this point in the history 
Setup JsDoc for API autogeneration
  • Loading branch information
@risenW
risenW authored Nov 25, 2020
2 parents 8033a72 + 9b34b8d commit 9b99aed
Show file tree
Hide file tree
Showing 8 changed files with 614 additions and 321 deletions.
269 changes: 1 addition & 268 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,19 +49,6 @@ file.rows() => stream object for rows
- [Browser](#browser)
- [Usage](#usage)
- [API](#api)
- [open](#open)
- [Files](#files)
- [load](#load)
- [Metadata](#metadata)
- [stream](#stream)
- [buffer](#buffer)
- [rows](#rows)
- [Datasets](#datasets)
- [load](#load-1)
- [addResource](#addresource)
- [Utilities](#utilities)
- [isDataset](#isdataset)
- [parseDatasetIdentifier](#parsedatasetidentifier)
- [Developers](#developers)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->
Expand Down Expand Up @@ -160,264 +147,10 @@ Dataset.load(path).then(dataset => {
})
```

---

## API

### open

Load a file from a path or descriptor.

`
load(pathOrDescriptor, {basePath, format}={})
`

There are 3 types of file source we support:

* Local path
* Remote url
* Inline data

```javascript
const data = require('frictionless.js')

const file = data.open('/path/to/file.csv')

const file = data.open('https://example.com/data.xls')

// loading raw data
const file = data.open({
name: 'mydata',
data: { // can be any javascript - an object, an array or a string or ...
a: 1,
b: 2
}
})

// Loading with a descriptor - this allows more fine-grained configuration
// The descriptor should follow the Frictionless Data Resource model
// http://specs.frictionlessdata.io/data-resource/
const file = data.open({
// file or url path
path: 'https://example.com/data.csv',
// a Table Schema - https://specs.frictionlessdata.io/table-schema/
schema: {
fields: [
...
]
}
// CSV dialect - https://specs.frictionlessdata.io/csv-dialect/
dialect: {
// this is tab separated CSV/DSV
delimiter: '\t'
}
})
```

`basePath`: use in cases where you want to create a File with a path that is relative to a base directory / path e.g.

```
const file = data.open('data.csv', {basePath: '/my/base/path'})
```

Will open the file: `/my/base/path/data.csv`

This functionality is primarily useful when using Files as part of Datasets where it can be convenient for a File to have a path relative to the directory of the Dataset. (See also Data Package and Data Resource in the Frictionless Data specs).


### Files

A single data file - local or remote.

#### load

*DEPRECATED*. Use simple `open`.

#### Metadata

Main metadata is available via the `descriptor`:

```
file.descriptor
```

This metadata is a combination of the metadata passed in at File creation (if you created the File with a descriptor object) and auto-inferred information from the File path. This is the info that is auto-inferred:

```
path: path this was instantiated with - may not be same as file.path (depending on basePath)
pathType: remote | local
name: file name (without extension)
format: the extension
mediatype: mimetype based on file name and extension
```

In addition to this metadata there are certain properties which are computed on demand:

```javascript
// the full path to the file (using basepath)
const path = file.path

const size = file.size

// md5 hash of the file
const hash = file.hash()

// sha256 hash of the file
const hash256 = file.hash(hashType='sha256')

// file encoding
const encoding = file.encoding
```

**Note**: size, hash are not available for remote Files (those created from urls).


#### stream

`stream()`

Get readable stream

@returns Promise with readable stream object on resolve

#### buffer

`File.buffer`

Get this file as a buffer (async)

@returns: promise which resolves to the buffer
See API documentation [here](./docs)

#### rows

`rows({keyed}={})`

Get the rows for this file as a node object stream (assumes underlying data is tabular!)

@returns Promise with rows as parsed JS objects (depends on file format)

* `keyed`: if `false` (default) returns rows as arrays. If `true` returns rows as objects.

TODO: casting (does data get cast automatically for you or not ...)

**What formats are supported?**

The rows functionality is currently available for CSV and Excel files. The Tabular support incorporates supports for Table Schema and CSV Dialect e.g. you can do:

```javascript

// load a CSV with a non-standard dialect e.g. tab separated or semi-colon separated
const file = data.open({
path: 'mydata.tsv'
// Full support for http://specs.frictionlessdata.io/csv-dialect/
dialect: {
delimiter: '\t' // for tabs or ';' for semi-colons etc
}
})

// open a CSV with a Table Schema
const file = data.open({
path: 'mydata.csv'
// Full support for Table Schema https://specs.frictionlessdata.io/table-schema/
schema: {
fields: [
{
name: 'Column 1',
type: 'integer'
},
...
]
}
})
```


### Datasets

A collection of data files with optional metadata.

Under the hood it heavily uses Data Package formats and it natively supports Data Package formats including loading from `datapackage.json` files. However, it does not require knowledge or use of Data Packages.

A Dataset has four primary properties:

* `descriptor`: key metadata. The descriptor follows the Data Package spec
* `resources`: an array of the Files contained in this Dataset
* `identifier`: the identifier encapsulates the location (or origin) of this Dataset
* `readme`: the README for this Dataset (if it exists). The readme content is taken from the README.md file located in the Dataset root directory, or, if that does not exist from the `readme` property on the descriptor. If neither of those exist the readme will be undefined or null.

In addition we provide the convenience attributes:

* `path`: the path (remote or local) to this dataset
* `dataPackageJsonPath`: the path to the `datapackage.json` for this Dataset (if it exists)

#### load

To create a new Dataset object use `Dataset.load`. It takes descriptor Object or identifier string:

```
async Dataset.load(pathOrDescriptor, {owner = null} = {})
```

* `pathOrDescriptor` - can be one of:
* local path to Dataset
* remote url to Dataset
* descriptor object
* @returns: a fully loaded Dataset (parsed and used `datapackage.json` and `README.md` -- if README exists)

For example:

```javascript
const data = require('frictionless.js')

const pathOrDescriptor = 'https://raw.githubusercontent.com/datasets/co2-ppm/master/datapackage.json'
const dataset = await data.Dataset.load(pathOrDescriptor)
```

#### addResource

Add a resource to the Dataset:

```javascript
addResource(resource)
```

* `resource`: may be an already instantiated File object or it is a resource descriptor
* @returns: null


### Utilities

#### isDataset

```javascript
// seeks to guess whether a given path is the path to a Dataset or a File
// (i.e. a directory or datapackage.json)
data.isDataset(path)
```

#### parseDatasetIdentifier

```javascript
// parses dataset path and returns identifier dictionary
// handles local paths, remote URLs as well as DataHub and GitHub specific URLs
// (e.g., https://datahub.io/core/finance-vix or https://github.com/datasets/finance-vix
const identifier = data.parseDatasetIdentifier(path)

console.log(identifier)
```

and it prints out:

```
{
name: <name>,
owner: <owner>,
path: <path>,
type: <type>,
original: <path>,
version: <version>
}
```

## Developers

Expand Down
12 changes: 6 additions & 6 deletions dist/browser/bundle.js
View file

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion dist/browser/bundle.js.map
View file

Large diffs are not rendered by default.

44 changes: 12 additions & 32 deletions dist/node/file-base.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
Object.defineProperty(exports, "__esModule", {
value: true
});
exports.computeHash = computeHash;
exports.File = void 0;

var _data = require("./data");
Expand All @@ -27,15 +26,6 @@ const {
} = require('stream');

class File {
constructor(descriptor, {
basePath
} = {}) {
this._descriptor = descriptor;
this._basePath = basePath;
this._descriptor.encoding = this.encoding || _data.DEFAULT_ENCODING;
this._computedHashes = {};
}

static load(pathOrDescriptor, {
basePath,
format
Expand All @@ -47,6 +37,14 @@ class File {
});
}

constructor(descriptor, {
basePath
} = {}) {
this._descriptor = descriptor;
this._basePath = basePath;
this._descriptor.encoding = this.encoding || _data.DEFAULT_ENCODING;
}

get descriptor() {
return this._descriptor;
}
Expand Down Expand Up @@ -101,22 +99,8 @@ class File {
})();
}

async hash(hashType = 'md5', progress, cache = true) {
if (cache && hashType in this._computedHashes) {
if (typeof progress === 'function') {
progress(100);
}

return this._computedHashes[hashType];
} else {
let hash = await computeHash(await this.stream(), this.size, hashType, progress);

if (cache && this != null) {
this._computedHashes[hashType] = hash;
}

return hash;
}
async hash(hashType = 'md5', progress) {
return _computeHash(await this.stream(), this.size, hashType, progress);
}

async hashSha256(progress) {
Expand Down Expand Up @@ -191,18 +175,14 @@ class File {

exports.File = File;

function computeHash(fileStream, fileSize, algorithm, progress, encoding = 'hex') {
function _computeHash(fileStream, fileSize, algorithm, progress) {
return new Promise((resolve, reject) => {
let hash = _crypto.default.createHash(algorithm);

let offset = 0;
let totalChunkSize = 0;
let chunkCount = 0;

if (!['hex', 'latin1', 'binary', 'base64'].includes(encoding)) {
throw new Error(`Invalid encoding value: ${encoding}; Expecting 'hex', 'latin1', 'binary' or 'base64'`);
}

const _reportProgress = new Transform({
transform(chunk, encoding, callback) {
if (chunkCount % 20 == 0) {
Expand All @@ -226,7 +206,7 @@ function computeHash(fileStream, fileSize, algorithm, progress, encoding = 'hex'
chunkCount += 1;
hash.update(chunk);
}).on('end', function () {
hash = hash.digest(encoding);
hash = hash.digest('hex');

if (typeof progress === 'function') {
progress(100);
Expand Down
6 changes: 0 additions & 6 deletions dist/node/index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,6 @@ Object.defineProperty(exports, "File", {
return _fileBase.File;
}
});
Object.defineProperty(exports, "computeHash", {
enumerable: true,
get: function () {
return _fileBase.computeHash;
}
});
Object.defineProperty(exports, "FileInterface", {
enumerable: true,
get: function () {
Expand Down
Loading

0 comments on commit 9b99aed

Please sign in to comment.