Data Requirements

By utilizing process intelligence, we are looking to analyze specific events in a chain of structured business activities. The events typically change the state of data and/or a product and generate some type of output.

Some examples of business processes include receiving orders, invoicing, shipping products, updating employee information, or servicing a customer. Business processes occur at all levels of an organization’s activities and include events that the customer sees and events that are hidden in information systems. The term also refers to the mix of all the separate steps toward the final business goal.

What kind of data does ABBYY Timeline require?

ABBYY Timeline assumes the existence of an event log where each event refers to a case, an activity, and a point in time. An event log can be seen as a collection of cases and a case can be seen as a trace/sequence of events.

Event data may come from a wide variety of sources:

  • a database system (e.g., patient data in a hospital)
  • a comma-separated values (CSV) file or spreadsheet
  • a transaction log (e.g., a trading system)
  • a business suite/ERP system (SAP, Oracle, etc.)
  • a message log (e.g., from IBM middleware)

Example Data Set

Data File Requirements for TimelinePI

The most common mechanism to load your data into the ABBYY Timeline platform is via a CSV file. This document describes the required structure of the file, along with some tips and tricks related to the generation of the file.

File Structure

The data should be placed into a comma-separated file. Each row in the file represents an event – a record that something happened to a specific object at a particular time. The file must have three mandatory columns and can include any number of optional columns. All columns can have arbitrary names, as no naming rules are imposed, however no two columns should have the same name.

The mandatory columns are:

  • TimelineID – A column for some identifier of an object you want to track over time. This could be an Order ID, Claim ID, Patient Encounter Number, Support Ticket Number, and so on.
  • Timestamp – A column for the timestamp showing when something happened in the life of the object. This column generally contains a date and time (see format description below). If a date with no time is provided, midnight (12:00 AM, 00:00:00) will be used.
  • Event Name – A column describing what happened to the object at that time – Order Submitted, Patient Departed, Adjuster Assigned, Ticket Escalated etc.

Optional columns

In addition to the mandatory columns, the file can contain any number of additional columns which will be used as dimensional attributes. You can filter by these fields, group and break down by them, or use them as additional information when analyzing the processes.

Order of Records

The order of the records in the file doesn’t matter, with one exception. If several records, related to the same object, have the exact same timestamp, the application will keep them in the same order they were placed in the file.

File Sample

[table id=2 /]

Here, A and B are the identifiers of the traceable objects, in our example – Students. Event names are “Student Applied, Student Accepted” etc.

In a CSV file, it looks like this:

TimelineID,Timestamp,Event name,Employee,Location A,1/16/2017 7:20:15,Student Applied,John,Boston A,3/10/2017 16:54:10,Student Accepted,Mary,Boston A,4/11/2017 15:04:00,Bill Generated,Ann,Charlotte B,2/1/2017 9:15:00,Student Applied,John,Boston B,3/2/2017 16:20:05,Student Accepted,Mary,Boston

Common Issues

The most common issues with data files are a bad date/time format and a broken file format.

Date/Time Format Issues

  • 1/6/2017 7:20:15
  • 1/6/2017 7:20:15 AM
  • 2017-01-06 7:20:15
  • 2017-01-06T7:20:15Z

Make sure you save timestamps in the file in one of these formats. Keep in mind, the default Time Format in Excel doesn’t include seconds. In order to save seconds into a CSV file, switch to Custom Formats in Excel, select “m/d/yyyy h:mm” and change it to “m/d/yyyy h:mm:ss”.

File Format Issues

If the values in any field include commas, the format of the file may break. To avoid this, make sure to specify double quotes as the string qualifier. Excel does it automatically, however, some tools like MS SQL Export Wizard require manual settings like the ones seen below.

The file should be Locale English (United States) and US ASCII or UTF-8 encoded.