Dataset Update Solution Using Files

Go to the following link for the latest and complete usage guide

https://api.guandata.com/apidoc/project-345092/api-3470542

---

Paths and Files

1. Upload files to the following server directories:

/home/guandata/data/guandata-store/upload_dataset (non-MinIO environment)

/home/guandata/data/minio/guandata-store/upload_dataset (MinIO environment)

2. Each dataset is stored as a directory under this directory. The directory name is the dataset ID, namely dsId.

File Format

1. Save as parquet files with snappy compression. Write a file group, not one large file.

2. Escape the following special characters in field names:
   ' ', ',', ';', '{', '}', '(', ')', '\n', '\t', '=', '.'

The selected escape character is %, so % must also be escaped. Scala sample code:

 private val forbiddenParquetColumNameChars = Set(' ', ',', ';', '{', '}', '(', ')', '\n', '\t', '=', '.')
  
 def encodeParquetColumnName(name: String) = name.map { x => {
   if (forbiddenParquetColumNameChars.contains(x) || x == '%') {
     f"%%${x.toInt}%02x"
   } else x.toString
 }
 }.mkString

Data Update API

After copying the data files to the specified path, call the following API to notify the BI service to update the dataset.

URL

$home_url/public-api/data-source/{dsId}/refresh-with-file

METHOD

GET

Header

Add x-auth-token. This token is the token after the user logs in to the system. For the login API, see section 1.1 of the following document:

https://api.guandata.com/apidoc/project-345092/doc-338136

PARAMETERS

Name	Location	Type	Meaning	Required	Remarks
dsId	Path	String	Dataset id	Yes
overwrite	Query	Boolean	Whether to perform a full update	Yes	true: full; false: incremental
fileType	Query	String	File type	No	Currently only PARQUET is supported

Response

{
  "result":"ok",  // "ok": API call succeeded (does not mean the dataset update succeeded); "fail": failed
  "response":     // Returns generated dataset update task information on success; null on failure
  {
    "taskId":"9ed75970-4439-11eb-a411-93e5c8edcd0e",   // Generated dataset update task id
"status":"Submitted",
"result":"Processing"
  },
"error":                                            // Returned on failure
  {
    "status": 500,                                    // Error code
"message": "Submission failed",                   // Error message
"detail": {}                                      // Error details
  }
}