Ingesting Data from Web Service

Overview

Feature Description

Ingesting data from a Web Service means Guandata BI can use the Web Service connector to retrieve API data. You can flexibly configure parsing rules for API responses, select the required fields, and create a Web Service dataset from the connection.

Use Cases

Web Service is a network-based distributed computing technology that allows different applications to interoperate over a network. Through Web Service APIs, applications can access remote services and retrieve data, which can then be imported into BI for further processing.

User Guide

Create a Data Connection

Entry point: Data Preparation > Dataset > New Dataset > Application > Web Service.

First, fill in the Web Service URL, request method, headers, authentication, request parameters, request body, pre-operations (optional), and post-operations (optional). Click Send Request to test the connection and view the response, including the status code and response body.

Configuration Item	Description
Request Method	Supports GET and POST, with GET as the default. GET is typically used to request resources, while POST is typically used to submit resources.
Headers	Key/value pairs that tell the server what resource type is required. For details, see: https://www.cnblogs.com/yunlongaimeng/p/10904558.html
Authentication	Currently supports only two authentication methods: API Key and Token. API Key: a secret token submitted with a Web Service request to identify the request source. Token: a credential valid for a specified period that represents a user role and grants permission to call the API.
Request Parameters	The fields and conditions to retrieve. Global Parameters are supported. See Global Parameters for details. Users can add User Basic Attributes as parameters when creating a dataset and view the parameter details on the dataset details page. After configuration, the data visible in Dashboards can be controlled by these parameters, enabling more fine-grained data security and personalized data viewing.
Request Body	JSON only. Used to encapsulate request parameters for POST requests. GET requests do not have a request body.
Pagination Parameters	When an API returns data in pages, configure pagination parameters to retrieve the data. Currently, pagination based on page numbers and offsets is supported.
Pre-operations	Before each main API call, the configured pre-operation is executed first. Part of the pre-operation result can be stored as dynamic parameters and referenced in the main API request headers, request parameters, or request body to enable dynamic parameter retrieval.
Post-operations	Use custom JavaScript to intercept a Web Service response and process, transform, or filter the data. This can convert the response into a standard structure compatible with the platform's JSONPath parsing method before configuring data parsing. See the example below for usage details.

1. Select Pre-operation and click Add Pre-operation:

   

2. Fill in the required request URL information for the pre-operation. You can also add dynamic parameters based on the response:

   • Dynamic Parameters: Parse the response content through a result field path to obtain dynamic parameter values. This is commonly used to retrieve token information. Created dynamic parameters can be referenced in subsequent request parameters using ${parameter_name}. This is optional, so you can run a pre-operation without passing any parameters.
   • Result Field Path: Use JSONPath to parse the response content and obtain the target parameter value.

   With these settings, the authentication API can be used as a pre-operation, and its returned token can be passed as a dynamic parameter into the Web Service dataset configuration.

   

3. Click Confirm to save the pre-operation.

   

4. Based on the API documentation, use dynamic parameters in the Web Service dataset headers, request body, or request parameters to enable dynamic parameter passing.

   

Notes

Dynamic parameter values are not currently cached. As a result, the pre-operation is executed every time the Web Service dataset is refreshed.

You should therefore evaluate any rate limits on the business-side authentication API. Frequent calls may trigger those limits.

Request Parameters

When configuring request parameters, add a parameter name and parameter value. Parameter values can use time macros, parameters, or user basic attributes.

User Basic Attributes means users can add User Basic Attributes as parameters when creating a dataset, and they can also view specific parameter details on the dataset details page. After configuration, data visible in Dashboards can be controlled by these parameters, enabling finer-grained data security and personalized data viewing.

Notes

User Basic Attributes can also be configured in the Request Body.

Post-operations

Custom JavaScript can be used to intercept the Web Service response and process, transform, or filter the data. This converts it into a standard structure compatible with the platform's JSONPath parsing method before configuring data parsing.

Example:

Define the convertJson method and return the modified response:

function convertJson(response) {
  // Define the response modification logic in this method
  return response; // Return the modified response to the BI platform
}

The following examples show the supported basic syntax using a simple response:

// Original response
{
  "a": 1
}

Scenario 1: Add a field

// Post-operation script
function convertJson(response) {
  response["b"] = 2;
  return response;
}

// Modified response
{
  "a": 1,
  "b": 2
}

Scenario 2: Modify a field

// Post-operation script
function convertJson(response) {
  response["a"] = 2;
  return response;
}

// Modified response
{
  "a": 2
}

Scenario 3: Delete a field

// Post-operation script
function convertJson(response) {
  delete response["a"];
  return response;
}

// Modified response
{
}

Scenario 4: Convert a response with separate structure and values into key-value pairs

// Post-operation script
function convertJson(response) {
response = response["data"];
var i_data = response["results"][0]["series"][0];
var o_data = [];
for(var i=0;i<i_data["values"].length;i++){
var temp = {}
for(var j=0;j<i_data["columns"].length;j++){
temp[i_data["columns"][j]] = i_data["values"][i][j] == null ? "null" : i_data["values"][i][j];
}
o_data.push(temp);
}
return o_data;
}

Configure Data Parsing

After the connection test succeeds, you can view the API response structure. By using standard JSONPath rules to parse JSON, you can enter a result field path. The system automatically parses all key/value pairs of objects directly under the first level of that path. If you need to further parse nested structures, refer to Parse JSON with ETL.

If you need to add a key/value pair from a nested object under the result field path, you can add it by creating a new field.

Finally, select the required fields, modify field names and types, and preview 30 rows of data.

Data Connection and Refresh Modes

Guandata BI supports two data connection modes:

• Direct Connection: Uses a RESTful API to retrieve near-real-time interface data and display it in BI visualizations. API parameterized queries can also be implemented with features such as Global Parameters. In this mode, the system does not persist extracted data or perform incremental updates, so the data remains real-time.
• Extract: Uses a RESTful API to perform a full historical data extract, together with an incremental strategy. The retrieved data is stored in the BI platform as an extracted dataset for subsequent processing.

This article does not go into detail about Direct Connection, Extract, or their related settings. For details, see Standard Database Connection Guide.

Notes

1. After Global Parameters are used in an API request, only Direct Connection is supported; Extract is not supported.
2. When the connection mode is Direct Connection, you can set a reasonable cache validity period for the Web Service dataset and choose whether to support real-time Card data.
3. SOAP is not supported.

Confirm Dataset Information

Specify the dataset name and storage location, then click Confirm Creation to complete dataset creation.

Edit the Dataset Model Structure

After creating the Web Service dataset, go to the dataset details page to view the overview, related Cards, and data security information. On the Model Structure page, click Edit in the upper-right corner to modify the dataset model structure. Editable items include:

• Request method
• Request URL
• Headers / authentication / request parameters / request body
• Result field path

Notes

For Web Service datasets of the Extract Data type (created with the Extract connection mode), parameters and user basic attributes cannot be used as parameter values.

Dataset Refresh Settings

On the dataset details page, click Data Refresh to modify incremental refresh settings. You can also manually trigger a dataset refresh in two ways: overwrite old data or append new data. Request parameters and request body values can also be customized and support dynamic time macros.

For more information about time macros and parameters, see Dynamic Time Macros and Global Parameters.

Notes

Incremental refresh can only be configured for datasets in Extract mode.

For error details, see Data Volume Exceeds the Allowed Threshold.

Parameter Entry Guide

Web Service supports Global Parameters, time macro parameters, user basic attribute parameters, and tokens in request parameters, the request body, and pre-operations. The following section explains how to fill in these parameters to help users get started quickly.

Category	Relevant Entry Points
Global Parameters	Request Parameters, Request Body, Pre-operations
Time Macros	Request Parameters > Parameter Value dialog, Request Body > Quick Add Time Macro, Pre-operations > Add/Edit Pre-operation
User Basic Attribute Parameters	Request Body > Add User Basic Attribute, Pre-operations > Add Pre-operation Parameter > Add User Basic Attribute

FAQ

Q: How can I test Web Service data ingestion for API data?

A: There are free APIs available for testing. You can register on your own and then use the free API quota. Examples:

• Weather API (GET): https://www.kancloud.cn/ccjin/yingq/603579
• 24 Solar Terms (POST): https://www.apishop.net/#/api/detail/?productID=88
• More API recommendations: https://www.zhihu.com/question/32225726