Smart ID Engine SDK Documentation

Smart ID Engine is a multi-platform, stand-alone SDK for recognizing and identifying text fields of passports, visas and other identification documents.

Workflow

The IdEngine workflow consists of the following stages:

Creating a recognition engine.
Setting session options.
Setting a recognition session.
Creating an image object.
Recognition process.
Processing the recognition result.
Memory deallocation.

Creating a Recognition Engine

From System Description:

Recognition Engine is the object where all recognition tools are stored and initialized. It is created using the appropriate configuration bundle containing all possible settings: a list of documents supported by a specific SDK, their fields, a list of possible authentication checks, and so on.

INFO

In special cases, the bundle is not supplied separately, but as part of (inside) the library.

Initialize the engine in a single instance. One instance allows you to create multiple recognition sessions. However, creating multiple instances of the engine is also possible.

The initialization process is a resource-intensive operation (as is the image analysis itself), so perform it off the main UI thread.

Create an IdEngine instance:

C++Java

C++

std::unique_ptr<se::id::IdEngine> engine(
    se::id::IdEngine::Create(configuration_bundle_path));

Java

import com.smartengines.id.*;

try {
    // load library
    System.loadLibrary("jniidengine");
    IdEngine instance = IdEngine.Create(configuration_bundle_path, true); 
} catch (Exception e) {
    Log.e("SMARTENGINES", e.getMessage());
}

Parameters:

configuration_bundle_path — the path to the onfiguration bundle file containing DocEngine settings (a .se file);
boolean, enabling/disabling lazy configuration (true by default). Optional.

Each delivery includes one or more bundles — archives files containing the settings required for creating objects and configuring Smart ID Engine.

Attention!

IdEngine::Create() is a factory method and returns an allocated pointer. The caller is responsible for deleting it.

Settings Session Options

From System Description:

Session settings is an object storing:

The list of the supported documents for recognition, grouped by internal engines. It is set in the configuration bundle with which the engine was created (read-only);
Advanced information about documents, including links to PRADO (read-only, used in Smart ID Engine only);
The list of documents submitted for recognition (* by default);
The list of expected fields for recognition (all by default);
The list of document sets (the mode parameter, default by default);
The special session options: the number of recognition threads, the expansion of the field list, the session timeout, and so on. You can find the full list of the options in our documentation.

Create an IdSessionSettings object using the configured IdEngine instance::

C++Java

C++

std::unique_ptr<se::id::IdSessionSettings> settings(engine->CreateSessionSettings());

Java

import com.smartengines.id.*;
IdSessionSettings settings = engine.CreateSessionSettings();

Attention!

IdEngine::CreateSessionSettings() is a factory method and returns an allocated pointer. The caller is responsible for deleting it.

Enabling Document Types

The document type is a string that describes the physical document to be recognized.

Each document type transmitted to the system is mapped to all supported types. If a certain type is supported, it is added to the list of recognized documents.

For example, the deu.passport.type1 type matches *deu.*, passport and *.

C++Java

C++

// Enabling all the supported documents of Germany
settings->AddEnabledDocumentTypes("deu.*");

Java

// Enabling all the supported documents of Germany
settings.AddEnabledDocumentTypes("deu.*");

To exclude document types from the recognition process, use the DocumentTypes(...) method.

TIP

To improve the processing quality, specify the minimum number of recognized document types.

Supported Document Types

Get the document types Smart Document Engine SDK can recognize:

C++Java

C++

// Iterating through the found internal engines
for (auto engine_it = settings->InternalEngineNamesBegin();
     engine_it != settings->InternalEngineNamesEnd();
     ++engine_it) {  
    // Iterating through supported document types supported by the engine 
    for (auto it = settings->SupportedDocumentTypesBegin(engine_it.GeValue());
         it != settings->SupportedDocumentTypesEnd(engine_it.GetValue());
         ++it) {
        // the it.GetValue() method returns the documents types inside the  
        // engine_it.GetValue() engine 
    }
}

Java

// Iterating through the found internal engines
for (StringsSetIterator engine_it = settings.InternalEngineNamesBegin();
     !engine_it.Equals(settings.InternalEngineNamesEnd());
     engine_it.Advance()) {
    // Iterating through supported document types supported by the engine
    for (StringsSetIterator it = settings.SupportedDocumentTypesBegin(engine_it.GeValue());
         !it.Equals(settings.SupportedDocumentTypesEnd(engine_it.GetValue()));
         it.Advance()) {
        // the it.GetValue() method returns the document types inside the engine_it.GetValue() engine
    }
}

Modes

From System Description:

Internal engines are groups of documents organized in such a way that their search and recognition algorithm works as efficiently as possible.

The list of documents for recognition is generated within the configuration bundle. Inside it, documents can be grouped by internal engines. For example:

All the documents of one country;
The passports of all countries;
All the ID documents of the CIS countries and so on.

Bundles may contain multiple documents located in different internal engines. If such documents are specified in the session settings, the system will not be able to determine independently which internal engine to select.

To solve this problem, the mode parameter is specified for each internal engine. Create a session setting this parameter with the list of recignized documents.

Attention!

You can specify multiple document types in one session only if they belong to the same internal engine. In other words, one recognition session can only work with one internal engine.

You can obtain information about the internal engines, their set and modes, as well as in the documentation for each specific SDK from the session settings.

Use the IdSessionSettings class to get a list of available modes in the configuration bundle:

C++Java

C++


for (auto it = settings->SupportedModesBegin(); 
    it != settings->SupportedModesEnd(); ++it) {
    // it.GetValue() the supported mode name
}

Java

for (StringsSetIterator it = settings.SupportedModesBegin();
     !it.Equals(settings.SupportedModesEnd());
     it.Advance()) {
    // it.GetValue() the supported mode name
}

By default the default mode is used. To get the current mode and set a new one, use the methods:

C++Java

C++

std::string current_mode = settings->GetCurrentMode();
// Setting the AnyPassport mode
settings->SetCurrentMode("anypassport");

Java

String current_mode = settings.GetCurrentMode();
// Setting the AnyPassport mode
settings.SetCurrentMode("anypassport");

The following recognition modes are available:

Mode	Recognized document types
anydoc	all documents
anypassport	international passport of all the world
anyid	ID documents of all the world
anydrvlic	driver's licenses of all the world

You can use the singleshot mode to recognize a single frame.

In mobile builds, it is recommended to use it to recognize images from the gallery.

You can also use the singleshot mode for any modes, such as anydoc-singleshot or anypassport-singleshot. Thesingleshot mode more resource consumptable, but more resistant to various non-mobile use cases.

You can override recognized document types.

Setting a Recognition Session

You can set a specific set of options for each recognition session using the methods of the IdSessionSettings class.

To get the options names and values:

C++Java

C++

for (auto it = settings->OptionsBegin();  
     it != settings->OptionsEnd();  
     ++it) {  
    // it.GetKey() returns the option name  
    // it.GetValue() returns the option value 
}

Java

for (StringsMapIterator it = settings.OptionsBegin();  
     !it.Equals(settings.OptionsEnd());  
     it.Advance()) {  
    // it.GetKey() returns the option name   
    // it.GetValue() returns the option value  
}

To change the option values, use the method SetOption(...) method:

C++Java

C++

settings->SetOption("common.extractTemplateImages", "true"); // Getting the template images
settings->SetOption("common.sessionTimeout", "3.0");

Java


settings.SetOption("common.extractTemplateImages", "true"); // Getting the template images
settings.SetOption("common.sessionTimeout", "3.0");

Spawning a Session

From System Description:

A personal signature is provided to the customer with the product. It is contained in the README.html file in the /doc directory.

Each time an instance of the recognition session is created, the signature must be passed as one of the arguments. This confirms the caller's right to use the library and unlocks it.

Signature is verified offline. The library does not access any external resources.

Spawn a session (the IdSession object):

C++Java

C++

const char* signature = "... YOUR SIGNATURE HERE ..."; // Your personal signature you use to start Smart ID Engine session

std::unique_ptr<se::id::IdSession> session(
    engine->SpawnSession(*settings, signature, &my_feedback));

Java

import com.smartengines.id.*;
String signature = "... YOUR SIGNATURE HERE ..."; // Your personal signature you use to start Smart ID Engine session
  
IdSession session = engine.SpawnSession(settings, signature, my_feedback);

Creating an Image Object

From System Description:

Pass an image of the special class se.common.image to the system for recognition. You can create it using the following image formats:

jpeg, png;
tiff (✔️TIFF_LZW, ✔️TIFF_PACKBITS,✔️TIFF_CCITT);
base64 (above mentioned formats);
file buffer with a preliminary indication of the color scheme, width\height\number of channels.

The maximum allowed image size by default is 15000x15000px. You can change the maximum image size.

Handling a HEIC file

A HEIC file in the mobile SDK are handled similarly to other image formats. The HEIC is read using system tools.

In the server SDK, open the HEIC format using external tools and convert it either to one of the formats we support, or transfer the raw pixels directly as an RGB buffer (this is recommended).

Create an Image object for further processing:

C++Java

C++

std::unique_ptr<se::common::Image> image(
    se::common::Image::FromFile(image_path)); // Loading from a file

Java

import com.smartengines.common.*;
Image image = Image.FromFile(image_path);

Information

To pass any images to the engine, an object of the se.common.Image class is required.

Actual methods:

Image.FromFile(imagePath) — takes the local path to the file for input;
Image.FromFileBuffer(data) — takes a file read into the buffer;
Image.FromBase64Buffer(data) — takes a file wrapped in base64 and read into the buffer;
Image.FromBufferExtended(raw_data, width, height, stride, pixel_format, bytes_per_channel) — creating an Image object using raw pixels;
Image.FromYUV(planeY, planeU, planeV, yuvDimensions) — creating an Image object from YUV_420_888.

Supported file formats:

jpeg, png;
tiff (✔️TIFF_LZW, ✔️TIFF_PACKBITS,✔️TIFF_CCITT);
base64 (above mentioned formats);
file buffer with a preliminary indication of the color scheme, width\height\number of channels.

The maximum allowed image size by default is 15000x15000px. You can change the maximum image size.

Handling a HEIC file

A HEIC file in the mobile SDK are handled similarly to other image formats. The HEIC is read using system tools.

In the server SDK, open the HEIC format using external tools and convert it either to one of the formats we support, or transfer the raw pixels directly as an RGB buffer (this is recommended).

Attention!

Image::FromFile() is a factory method and returns an allocated pointer. The caller is responsible for deleting it.

Recognition Process

Call the Process(...) method to process the document image:

C++Java

C++

const se::id::IdResult& result = session->Process(*image);

Java

import com.smartengines.id.*;  
IdResult result = session.Process(image);

To work with a video stream, you should transmit a series of images in a loop.

Дать пример с терминальностью

The exit from the cycle is automatic due to the terminality mechanism.

From System Description:

Recognition session is the process of recognizing a physical document, such as a specific passport, driver's license, and so on.

The process of recognizing a physical document means handling one or a series of images of the same document.

The system can recognize a document with high accuracy even using a single image. However, using the results of recognition of multiple images significantly improves quality. The system combines recognition results from different frames. This allows you to recognize documents in conditions of poor lighting, glare, and other adverse factors with a high confidence. The mechanism that decides whether to recognize additional frames or to stop the process is called terminality.

Terminality is automatic stop of the recognition process in video stream. It takes the true value in two cases:

If new frames are added the recognition result will not change;
The session timeout specified in the settings has been reached. The recognition session is created with settings prepared in advance. Sessions are independent of each other, what allows you to use multiple sessions simultaneously if necessary (using the same or different settings).

Attention!

Image::FromFile() is not a factory method. The returned IdResult object is not independent. Its lifetime does not exceed the IdSession object lifetime.

Processing the Recognition Result

From System Description:

Recognition result is an interface object that contains all the available information about the recognition result. Each parameter is output using a separate method, which allows you to use only the required data in integration. The set of the object fields depends on the type of the session in which recognition was performed, and this depends on the functionality of the product used.

When working with a video stream, process the frames sequentially within the same session. The processing time is unlimited, but it is recommended to perform processing until the true value of the result.GetIsTerminal()option is obtained. The text filelds IdResult values contain information about the recognition results:

C++Java

C++

// Text results
for (auto it = result.TextFieldsBegin(); it != result.TextFieldsEnd(); ++it) {  
    const se::id::IdTextField& field = it.GetValue();

    bool is_accepted = field.GetBaseFieldInfo().GetIsAccepted(); // Getting the is_accepted flag value
    std::string field_value = field.GetValue().GetFirstString().GetCStr(); // Getting a field recognition result as a UTF-8 encoded string
}

// Images
for (auto it = result.ImageFieldsBegin(); it != result.ImageFieldsEnd(); ++it) {
    const se::id::IdImageField& field = it.GetValue();

    const se::common::Image& image = field.GetValue();
}

Java

import com.smartengines.id.*;

// Text results
for (IdTextFieldsMapIterator it = result.TextFieldsBegin();
     !it.Equals(result.TextFieldsEnd());
     it.Advance()) {
    IdTextField field = it.GetValue();

    boolean is_accepted = field.GetBaseFieldInfo().GetIsAccepted(); // Getting the is_accepted flag value

    String field_value = field.GetValue().GetFirstString().GetCStr(); // Getting a field recognition result as a UTF-8 encoded string
}

// Images
for (IdImageFieldsMapIterator it = result.ImageFieldsBegin();
    !it.Equals(result.ImageFieldsEnd());
     it.Advance()) {
    IdImageField field = it.GetValue();

    Image image = field.GetValue();
}

To check the result availability after session completion (terminality), we recommend to use the result.GetDocumentType() method. The return value is available throughout the entire session, provided that recognition has been performed at least once.

We recommend to use the result.GetTemplateDetectionResulsCount() method to analyze intermediate recognition results, because it allows you to check document availability in the current frame.

Callbacks

From System Description:

Recognition in video stream is a complex and often not quick process that requires user interaction. Providing information about the progress of recognition on the screen during operation, we help the user to understand the system operation stages better. For this purpose machine vision technologies are often complemented by augmented reality elements, what enhances the user experience and makes interaction more intuitive.

The recognition result object contains information about the coordinates of the quadrilateral of the document template (document borders) and the coordinates of the quadrilaterals of the document field templates. For example, using these coordinates, you can draw the found document borders on the screen using your camera preview.

Smart ID Engine SDK supports additional callbacks during the analysis of objects before their processing, the Process(...)method, is completed. This allows the user to get more information about the basic recognition process, and also helps to create an interactive graphical interface. To implement callbacks, create a subclass of the IdFeedback class and implement the following methods:

C++Java

C++

class MyFeedback : public se::id::IdFeedback {
public:
  virtual ~OptionalFeedback() override = default;

public:  
  virtual void FeedbackReceived(  
      const se::id::IdFeedbackContainer& /*feedback_container*/) override { }  
  virtual void TemplateDetectionResultReceived(  
      const se::id::IdTemplateDetectionResult& result) override { }  
  virtual void TemplateSegmentationResultReceived(  
      const se::id::IdTemplateSegmentationResult& /*result*/) override { }  
  virtual void ResultReceived(const se::id::IdResult& result) override { }  
  virtual void SessionEnded() override { }  
};

Java

import com.smartengines.id.*;
class MyFeedback extends IdFeedback {
  public void FeedbackReceived(IdFeedbackContainer feedback_container) { }  
  public void TemplateDetectionResultReceived(IdTemplateDetectionResult result) { }  
  public void TemplateSegmentationResultReceived(IdTemplateSegmentationResult result) { }  
  public void ResultReceived(IdResult result) { }  
  public void SessionEnded() { }  
}

Use the TemplateDetectionResultReceived(...) and TemplateSegmentationResultReceived(...) methods для to display the document zones and field boundaries in the graphical interface during real-time video stream recognition.
Create an instance of the MyFeedback class and pass it to the session during initialization:

C++Java

C++

MyFeedback my_feedback;  
std::unique_ptr<se::id::IdSession> session(  
    engine->SpawnSession(*settings, signature, &my_feedback));

Java

import com.smartengines.id.*;  
MyFeedback my_feedback = new MyFeedback();  
IdSession session = engine.SpawnSession(settings, signature, my_feedback);

Memory Deallocation

Despite the fact that garbage collection is implemented and working, it is recommended to call the obj.delete() method for our API objects manually, since they are wrappers of memory allocated on the heap. The size of their heap is unknown to the garbage collector, which can lead to a delay in deleting objects and, consequently, high overall memory consumption.

Java

IdEngine engine = IdEngine.Create(config_path); // or any other object
// ...
engine.delete(); // гарантирует принудительное и немедленное освобождение обернутого объекта C++

Some Smart Code Engine SDK classes contain factory methods that return pointers to objects allocated on the heap.Such objects must be deleted.

TIP

In C++ for easy memory management and prevention of memory leaks, use smart pointers std::unique_ptr<T> or std::shared_ptr<T>.
In Java API remove garbage using the .delete() method.

JNI Specifics

Consider Java integration specifics..

Smart ID Engine SDK includes JNI automatically generated from the C++ interface using SWIG.

When using an auxiliary feedback object with the help of the IdFeedback subclass, make sure that its instance has the same scope as the IdSession. The reason for this is that our API does not contain a pointer to an instance of the feedback object. As a result, garbage collection is performed prematurely, which leads to a failure.

Java

// Disadvantage: the garbage collector may prematurely delete an instance of the feedback object.  
class MyDocumentRecognizer {  
    private IdEngine engine;  
    private IdSession session;

    private void InitializeSmartIdEngine() {  
        // ...  
        session = engine.SpawnSession(settings, signature, new MyFeedback());  
        // the feedback object can be deleted by the garbage collector because the / object is out of session
} 
    }

Java

// Advantage: access to the recognition session  
class MyDocumentRecognizer {  
    private IdEngine engine;  
    private IdSession session;  
    private MyGeedback feedback; // feedback has session's scope

    private void InitializeSmartIdEngine() {  
        // ...  
        feedback = new MyFeedback();  
        session = engine.SpawnSession(settings, signature, feedback);  
    }
}

Library Interface

Common Classes

Common classes, such as Point, OcrString, Image etc. are located in these::common в директории secommon.

Header	Description
<secommon/se_export_defs.h>	Contains export-related definitions of Smart Engines libraries
<secommon/se_exceptions_defs.h>	Contains the definition of exceptions used in Smart Engines libraries
<secommon/se_geometry.h>	Contains geometric classes and procedures (`Point`, `Rectangle`, etc.)
<secommon/se_image.h>	Contains image classes and procedures
<secommon/se_string.h>	Contains string-related classes (`MutableString`, `OcrString`, etc.)
<secommon/se_string_iterator.h>	Contain string iterator definition
<secommon/se_serialization.h>	Contains auxiliary classes related to object serialization (not used in Smart ID Engine)
<secommon/se_common.h>	an auxiliary header which simply includes all of the above

The similar Java API is contained in the com.smartengines.common module:

Java

import com.smartengines.common.*; // Import se::common classes

Main Classes

The main Smart Code Engine classes are located within these::id namespace in idengine the directory:

Header	Description
<idengine/id_document_info.h>	Provides information about the document type (textual document description)
<idengine/id_engine.h>	Contains the `IdEngine` class definition
<idengine/id_session_settings.h>	Contains the `IdSessionSettings` class definition
<idengine/id_ session.h>	Contains the `IdSession` class definition
<idengine/id_result.h>	Contains the `IdResult`, `IdTemplateDetectionResult` и `IdTemplateSegmentationResult` classes definition
<idengine/id_fields.h>	Contains definition of the classes representing the Smart ID Engine fields
<idengine/id_feedback.h>	Contains the `IdFeedback` interface and the related containers
<idengine/id_face_feedback.h>	Contains the `IdFaceFeedback` interface
<idengine/id_face_session_settings.h>	Contains the `IdFaceSessionSettings` class definition
<idengine/id_face_session.h>	Contains the `IdFaceSession` class definition
<idengine/id_face_result.h>	Contains the classes describing the face recognition result
<idengine/id_field_processing_session_settings.h>	Contains the `IdFieldProcessingSessionSettings` class definition
<idengine/id_field_processing_session.h>	Contains `IdFieldProcessingSessionSettings` class definition
<idengine/id_file_analysis_session.h>	Session checking the image for anomalies
<idengine/id_file_analysis_session_settings.h>	Session setting checking the image for anomalies

The same Java API classes are contained in the com.smartengines.id module:

Java


import com.smartengines.id.*; // Import se::id classes

Exceptions

The C++ API may throw se::common::BaseException exceptions in case of incorrect data entry, incorrect calls, and other errors. The following exception subclasses are supported:

Exception	Description
FileSystemException	Thrown if attempt to read data from a non-existent file is mane
InternalException	Thrown in case of an unknown error of an internal system component
InvalidArgumentException	Thrown if a method with invalid input parameters is called
InvalidKeyException	Thrown in case of access the container using an invalid or a non-existent key, or in case of access to a list using an invalid or out-of-range index
InvalidStateException	Thrown if a system error occurs due to an invalid state of the system objects
MemoryException	Thrown if an allocation is attempted with insufficient RAM
NotSupportedException	Thrown when trying to access a method which is not supported in the current library version or is not supported at all
Uninitialized Object Exception	Thrown in case of an attempt to access a non-existent or non-initialized object

If exceptions are thrown, user-friendly messages are displayed using the e.what() method.

TIP

se::common::BaseException is not a subclass of std::exception. interface does not have any dependency on the STL.

The Java API exceptions are wrapped in common class java.lang.Exception. Тhe exception type is included in the corresponding message text.

Smart ID Engine SDK Documentation ​

Workflow ​

Creating a Recognition Engine ​

Settings Session Options ​

Enabling Document Types ​

Supported Document Types ​

Modes ​

Setting a Recognition Session ​

Spawning a Session ​

Creating an Image Object ​

Recognition Process ​

Processing the Recognition Result ​

Callbacks ​

Memory Deallocation ​

JNI Specifics ​

Library Interface ​

Common Classes ​

Main Classes ​

Exceptions ​

Smart ID Engine SDK Documentation

Workflow

Creating a Recognition Engine

Settings Session Options

Enabling Document Types

Supported Document Types

Modes

Setting a Recognition Session

Spawning a Session

Creating an Image Object

Recognition Process

Processing the Recognition Result

Callbacks

Memory Deallocation

JNI Specifics

Library Interface

Common Classes

Main Classes

Exceptions