Data Warehouse Data Ingestion

Question

I how can I ingest data from yuuvis in a Data Warehouse in a performant way?

Is the rest API the only way to access the data structures?

How can I do a delta-load, i.e. only changed data since last data extraction?

Answer 1

Hello Marvin,

the REST API is the only offical way to access the dms-objects, yes. You could use the ResultService.getQueryResult or ResultService.getEsqlResult endpoints to select the desired objects. The endpoints support json and xml output.

The most performant way to access and transfer the data is surely to directly query the database. If you could tell me a little more in detail which data you want to transfer, I can give you some exemplary statements.

Best regards
Nicolai

Comments

We currently export an objecttype of $datentyp from schema via a PERL script, cutting the export in pieces of 10.000 and putting them together in a postprocessing task:
$enaio_server/rest-ws/service/result/query?type=$datentyp\\&limit=$limit\\&offset=$offset\\&fields=\\*\\*
This script runs over 1h.
If I get you right we can accomplish this task by the not officially supported way and directly query the database. (we will manage all implications of updates, performance, etc.)

I assume I can access the database. How do I identify the relevant table of „$datentyp“ and are the attributes „&fields“ extracted via PERL also fully available?