Graph Importer
The GraphImporter is a high-performance, declarative graph importer that uses a two-pass CSR-first architecture to bulk-load graph data from XML, CSV, and JSONL files into ArcadeDB.
It is designed for importing large datasets (millions of vertices and edges) with minimal memory overhead.
The importer is located in the integration module (com.arcadedb.integration.importer.graph.GraphImporter).
| Never run the importer against a database directory that an ArcadeDB Server (or any other process) already has open. It opens the database files directly, bypassing the server, and two processes writing to the same database at once can corrupt it. To load data into a database a server is serving, use the server’s remote protocol instead, or stop the server for the duration of the import. |
How It Works
The import runs in two passes:
-
Pass 1 — Process each data source once: create vertices with full properties, collect graph topology as compressed int arrays
-
Pass 2 — Create all edges from the in-memory topology using GraphBatch, one batch per edge type with bidirectional edges for full IN+OUT traversal
-
Vector graphs — Build the graph of every
LSM_VECTORindex on a type the import wrote to, so the index answers at index speed the momentrun()returns (see Loading into a type with a vector index)
The speculative background maintenance of the database’s indexes (the vector index’s inactivity rebuild) is held off for the whole run, not only while one of the batches is open: a stall between the two passes must not be read as "the load is over".
Command-Line Usage
java -cp arcadedb-integration-*.jar com.arcadedb.integration.importer.graph.GraphImporter \
<json-config-file> <database-path> [data-dir]-
json-config-file— Path to the JSON configuration file (see JSON Configuration below) -
database-path— Path where the database will be created (any existing database at this path is deleted) -
data-dir— Optional base directory for resolving relative file paths in the JSON config (defaults to the JSON file’s parent directory)
Java API
The importer can also be used programmatically via a fluent Builder API:
try (GraphImporter importer = GraphImporter.builder(database)
.vertex("User", new CsvRowSource("users.csv"), v -> {
v.id("Id");
v.intProperty("reputation", "Reputation");
v.property("name", "DisplayName");
})
.vertex("Question", new XmlRowSource("posts.xml"), v -> {
v.id("Id");
v.filter("PostTypeId", "1");
v.property("title", "Title");
v.edgeIn("OwnerUserId", "ASKED", "User");
v.splitEdge("Tags", "TAGGED_WITH", "Tag", "|");
})
.edgeSource("LINKED_TO", new CsvRowSource("links.csv"), e -> {
e.from("PostId", "Question");
e.to("RelatedId", "Question");
e.intProperty("linkType", "LinkTypeId");
})
.limit(10000)
.build()) {
importer.run();
System.out.printf("Vertices: %,d, Edges: %,d%n",
importer.getVertexCount(), importer.getEdgeCount());
}Or from a JSON configuration file:
String json = new String(Files.readAllBytes(jsonFile.toPath()));
JSONObject config = new JSONObject(json);
GraphImporter.createSchemaFromConfig(database, config);
try (GraphImporter importer = GraphImporter.fromJSON(database, config, dataDir)) {
importer.run();
}
GraphImporter.executePostImportCommands(database, config);JSON Configuration
The JSON configuration file defines vertex types, edge types, data sources, property mappings, and optional post-import commands.
Vertex Definitions
Each entry in the vertices array defines a vertex type and its data source:
| Key | Required | Description |
|---|---|---|
|
Yes |
ArcadeDB vertex type name (auto-created if it does not exist) |
|
Yes |
Source file path, relative to the data directory. Format is auto-detected from extension: |
|
Yes |
Source attribute used as the primary key for edge resolution between types. The value is taken as
written, so an integer, a value too large for an integer, and a string such as |
|
No |
Secondary key, used by |
|
No |
Row filter in the format |
|
No |
For XML files: element name to read (defaults to |
|
No |
Maps ArcadeDB property names to source attributes, optionally with a type prefix. See Property Types |
|
No |
Array of edge definitions derived from foreign key attributes in this vertex’s source file |
Property Types
A property mapping is "dbPropertyName": "sourceAttribute", where the source attribute may carry a type prefix.
Without a prefix the value is imported as a string.
The same prefixes are accepted in vertex properties and in edge-source properties.
Before 26.10.1 an edgeSources entry understood only int:, long: and double:, and quietly ignored any
other mapping. If you have a configuration that declares other property types on an edge source, those properties
are now actually imported.
|
An empty value means "not set" in every source format. Before 26.10.1 that was true only for CSV: the same
blank column stored an empty string when the row came from a JSONL or XML file, so IS NULL and mandatory-property
checks answered differently depending on which format the data had been exported to.
|
| Prefix | Stored type | Notes |
|---|---|---|
(none) |
|
Null or empty source values are skipped |
|
|
Missing or empty values default to |
|
|
Missing or empty values default to |
|
|
Missing or empty values default to |
|
|
|
|
|
Default format |
|
|
A dense float vector (an embedding). JSONL reads the JSON array natively; CSV and XML parse the textual form |
|
|
A generic array, e.g. a list of tags |
Every prefix has to be followed by the name of the source attribute to read: a value such as "int:" on its own, or a datetime: pattern missing one half of its FORMAT|attribute pair, is reported as a configuration error.
The same goes for a missing type, edge, target, attribute or file key: the message names the key, the source that declares it, and what the value is for. (Since v26.10.1: these were accepted and then matched nothing, so the import ran and the property was simply never filled in.)
Importing Vector Embeddings
Embedding pipelines usually emit JSONL with the vector inline as a JSON array:
{"id": 1, "title": "The Hitchhiker's Guide to the Galaxy", "tags": ["scifi", "comedy"],
"embedding": [-0.31142, 0.51346, -0.02326, -0.29451, 0.57215]}Declare the column with the vector: prefix and create the index afterwards with a post-import command:
{
"vertices": [
{
"type": "Book", "file": "books.jsonl", "id": "id",
"properties": {
"title": "title",
"tags": "list:tags",
"embedding": "vector:embedding"
}
}
],
"postImportCommands": [
{ "language": "sql",
"command": "CREATE INDEX ON Book (embedding) LSM_VECTOR METADATA { dimensions: 5, similarity: 'COSINE' }" }
]
}The property is stored as a primitive float[], not as a list of boxed numbers: a 768-dimension embedding costs
roughly 3KB per vertex instead of 18KB, which matters on a bulk load of millions of rows, and it is the exact
representation the vector index consumes, so building the index converts nothing.
Use vector: only for numeric arrays. list: keeps the array as a generic list and is the right choice for
heterogeneous or non-numeric arrays such as tags.
The equivalent Java API calls are floatArrayProperty(name, attribute) and listProperty(name, attribute).
Loading into a type with a vector index
The index can also exist before the import. Every vector the load writes goes into the index’s delta buffer, where it is searchable by a linear scan, and the HNSW graph that makes the search fast is built afterwards. Left to the index, that build starts once the index has been quiet for its inactivity window, on a background thread - and a database.close() right after run(), which is what a loader normally does, cancels it and throws the work away.
So the importer builds it itself: when run() returns, every LSM_VECTOR index on a type the import wrote to has its graph built and persisted, and the process can close the database at once. The build is logged with its progress and takes as long as CREATE INDEX would over the same vectors. An index whose graph is already current is skipped.
To leave the build to the index’s background rebuild instead - only worth it when the database stays open long enough for that build to complete - set "vectorGraphBuild": false in the JSON configuration, or .withVectorGraphBuild(false) on the builder.
Arrays in a CSV file
CSV and XML have no array of their own, so an array column is read from its textual form [0.1,0.2,0.3].
In a CSV file, choose a delimiter the array does not contain - ; rather than the default , - because the
CSV reader splits on the delimiter without honouring quotes, and a comma-delimited file would cut the array
across several columns:
Id;Title;Embedding 1;The Hitchhiker's Guide to the Galaxy;[-0.31142,0.51346,-0.02326]
If the array is split anyway, the import stops and says so, naming the property and the delimiter.
Edge Definitions (within a vertex)
Each entry in a vertex’s edges array defines how to create edges from foreign key attributes:
| Key | Required | Description |
|---|---|---|
|
Yes |
Source attribute containing the foreign key value |
|
Yes |
ArcadeDB edge type name (auto-created if it does not exist) |
|
Yes |
Target vertex type the foreign key references |
|
No |
|
|
No |
Delimiter for multi-value fields (e.g., |
Edge-Only Sources
The edgeSources array defines edges where both endpoints already exist as vertices. No vertices are created from these sources:
| Key | Required | Description |
|---|---|---|
|
Yes |
ArcadeDB edge type name |
|
Yes |
Source file path |
|
Yes |
Compact format |
|
Yes |
Compact format |
|
No |
Property mappings (same format as vertex properties) |
General Options
| Key | Required | Description |
|---|---|---|
|
No |
Maximum records per source (for testing). Omit or set to 0 for unlimited |
|
No |
Whether to build the graph of every |
Post-Import Commands
The postImportCommands array defines commands to execute automatically after the graph import completes.
This is useful for creating indexes, analytical views, or running any database command that depends on the imported data being present.
| Key | Required | Description |
|---|---|---|
|
Yes |
Query language to use: |
|
Yes |
The command text to execute |
Commands are executed sequentially in the order they appear. If a command fails, a warning is logged and the remaining commands continue to execute.
If any post-import command triggers an asynchronous Graph Analytical View build, the importer automatically waits (up to 10 minutes) for all views to reach READY status before returning.
Example:
"postImportCommands": [
{
"language": "sql",
"command": "CREATE INDEX ON Question (Id) UNIQUE"
},
{
"language": "sql",
"command": "CREATE GRAPH ANALYTICAL VIEW IF NOT EXISTS myGraph PROPERTIES (`!Body`, `!Text`) UPDATE MODE SYNCHRONOUS"
}
]Complete Example
Below is a complete JSON configuration for importing a StackOverflow data dump:
{
"vertices": [
{
"type": "Tag", "file": "Tags.xml", "id": "Id", "nameId": "TagName",
"properties": { "Id": "int:Id", "TagName": "TagName", "Count": "int:Count" }
},
{
"type": "User", "file": "Users.xml", "id": "Id",
"properties": {
"Id": "int:Id", "DisplayName": "DisplayName", "Reputation": "int:Reputation",
"CreationDate": "CreationDate", "Views": "int:Views"
}
},
{
"type": "Question", "file": "Posts.xml", "id": "Id", "filter": "PostTypeId=1",
"properties": {
"Id": "int:Id", "Title": "Title", "Body": "Body",
"Score": "int:Score", "ViewCount": "int:ViewCount", "Tags": "Tags"
},
"edges": [
{ "attribute": "OwnerUserId", "edge": "ASKED", "target": "User", "direction": "in" },
{ "attribute": "Tags", "edge": "TAGGED_WITH", "target": "Tag", "split": "|" }
]
},
{
"type": "Answer", "file": "Posts.xml", "id": "Id", "filter": "PostTypeId=2",
"properties": {
"Id": "int:Id", "Body": "Body", "Score": "int:Score"
},
"edges": [
{ "attribute": "OwnerUserId", "edge": "ANSWERED", "target": "User", "direction": "in" },
{ "attribute": "ParentId", "edge": "HAS_ANSWER", "target": "Question", "direction": "in" }
]
}
],
"edgeSources": [
{
"edge": "ACCEPTED_ANSWER", "file": "Posts.xml",
"from": "Id:Question", "to": "AcceptedAnswerId:Answer"
},
{
"edge": "LINKED_TO", "file": "PostLinks.xml",
"from": "PostId:Question", "to": "RelatedPostId:Question",
"properties": { "LinkType": "int:LinkTypeId" }
}
],
"postImportCommands": [
{
"language": "sql",
"command": "CREATE GRAPH ANALYTICAL VIEW IF NOT EXISTS stackoverflow PROPERTIES (`!Body`, `!Text`) UPDATE MODE SYNCHRONOUS"
}
]
}Identities and Missing Edges
Every edge is resolved by matching the value of a foreign key against the id of the vertex type it
points at. The importer compares those values as text, so any key your files already use works
without being converted first:
{"id": "W13696992", "title": "A paper", "publication_year": 2008}from_id,to_id
2257721487,28031933
Before 26.10.1 both of these failed: an identity had to fit an int, so a string key or one
larger than about two billion aborted the import. Vertex ids of 0 were also unusable as an edge
endpoint. No configuration change is needed to benefit from the fix.
|
When an edge names a key that no vertex carries, the edge is skipped — there is nothing to attach it
to — but the importer now counts it and logs a warning per source, and the total is available from
getUnresolvedEdgeCount() in the Java API. A graph that comes out smaller than the files you fed in
no longer does so silently.
Four configurations that could only ever produce zero edges are now rejected with an error before anything is written, instead of importing a quietly incomplete graph:
-
an edge pointing at a vertex type that no source imports (usually a typo in the type name)
-
an edge pointing at a vertex type that is imported after the source referencing it. A vertex source can only resolve against types already imported, so declare the target’s source first
-
an edge resolving against the
idof a type that declares onlynameId, or the reverse -
two vertex sources declaring the same type name