Paginating Requests

The GDI exposes paging models that enable you to access large amounts of data across a number of smaller requests. Paging is configured by including the paging property in a query and configuring a combination of the pagination options described below. The GDI supports keyset-based, page-based, cursor-based, and offset-based pagination. Paging is supported for all data source types.

• Paging Syntax
• Paging Examples

Paging Syntax

s:paging [
  s:key (?variable) ;
  s:page ?variable ;
  s:cursor ?variable ;
  s:offset ?variable ;
  s:size int ;
  s:limit ?variable ;
] ;

Option	Type	Description
key	variable	Include this property if you want to configure keyset-based pagination where a key is specified to act as a delimiter of the page. The s:key value is a variable that is bound to an expression that defines how to delimit the data. It is usually calculated by an aggregate expression and/or filter that can be pushed to the source. The aggregate expression is typically MAX, but MIN can also be used to page through data in reverse order, such as when working with temporal data. See Key-Based Examples below for examples that configure paging using the s:key property.
page	variable	Include this property if you want to configure page-based pagination where the set is divided into pages. The s:page property value is a variable that the GDI can use to track the current page across requests. See Page-Based Example below for an example that configures paging using the s:page property.
cursor	variable	Include this property if you want to configure cursor-based pagination. The s:cursor property value is a variable that is bound against the source to capture the "cursor" value. The GDI uses this value as input to the source to deliver the next page of data. See Cursor-Based Example below for an example that configures paging with the s:cursor property.
offset	variable	Include this property along with the limit property if you want to configure offset-based pagination. The s:offset property value is a variable that the GDI can use to track the current offset across requests. See Offset-Based Example below for an example that configures paging using the s:offset property.
size	int	This property can be included with any of the paging models to configure the maximum size of each page. For example, s:size 5000 limits the page size to 5,000 rows.
limit	variable	This property can be included to define the variable that the GDI should use to push the page size back to the source.

Paging Examples

• Key-Based Examples
• Page-Based Example
• Cursor-Based Example
• Offset-Based Example

Key-Based Examples

The example SERVICE clause below pages data based on the ?LastID key, which is calculated by finding the maximum value of SalesOrderID and binding it to ?LastID. A FILTER is used to filter for data where the SalesOrderID is greater than ?LastID.

SERVICE <http://cambridgesemantics.com/services/DataToolkit>
{
  BIND(MAX(?SalesOrderID) AS ?LastID)
  FILTER(?SalesOrderID > ?LastID)

  ?SalesOrderHeaderEnlarged a s:DbSource ;
    s:url "jdbc:sqlserver://..." ;
    s:table "Sales.SalesOrderHeaderEnlarged" ;
    s:paging [
      s:key (?LastID) ;
      s:size 5000 ;
    ] ;
    ?SalesOrderID (xsd:int) ;
    ?RevisionNumber (xsd:int) ;
    ?OrderDate ("OrderDate" xsd:dateTime) ;
    ?DueDate (xsd:dateTime) .
}

The SERVICE clause below shows an example where key-based paging is configured to page through temporal data in reverse order. The s:limit property is configured on the s:HttpSource to limit the overall number of results returned across all pages. This query retrieves at most 1000 records (s:limit 1000), 100 rows (s:size 100) at a time.

SERVICE <http://cambridgesemantics.com/services/DataToolkit>
{
  BIND(MIN(?Timestamp) AS ?LastTimestamp)

  ?api a s:HttpSource ;
    s:url "http://slack.com/api/messages/latest" ;
    s:parameter [ s:name "before" ; s:value ?LastTimestamp ] ;
    s:parameter [ s:name "limit" ; s:value ?limit ] ;
    s:limit 1000 ;
    s:paging [
      s:key (?LastTimestamp) ;
      s:limit ?limit ;
      s:size 100 ;
    ] ;
    ?Message (xsd:string) ;
    ?Author (xsd:string) ;
    ?Timestamp (xsd:dateTime) .
}

Page-Based Example

The SERVICE clause below shows an example that uses the s:page property to configure page-based paging where the page size is 100 rows. This query retrieves at most 1000 records (s:limit 1000), 100 rows (s:size 100) at a time.

SERVICE <http://cambridgesemantics.com/services/DataToolkit>
{
  ?api a s:HttpSource ;
    s:url "http://slack.com/api/messages" ;
    s:parameter [ s:name "page" ; s:value ?page ] ;
    s:parameter [ s:name "size" ; s:value ?limit ] ;
    s:limit 1000 ;
    s:paging [
      s:page ?page ;
      s:limit ?limit ;
      s:size 100 ;
    ] ;
    ?Message (xsd:string) ;
    ?Author (xsd:string) ;
    ?Timestamp (xsd:dateTime) .
}

Cursor-Based Example

The SERVICE clause below shows an example that uses the s:cursor property to configure cursor-based paging.

SERVICE <http://cambridgesemantics.com/services/DataToolkit>
{
  ?api a s:HttpSource ;
    s:url "http://slack.com/api/messages" ;
    s:parameter [ s:name "cursor" ; s:value ?cursor ] ;
    s:parameter [ s:name "limit" ; s:value ?limit ] ;
    s:limit 1000 ;
    s:paging [
      s:cursor ?cursor ;
      s:limit ?limit ;
      s:size 100 ;
    ] ;
    ?Message (xsd:string) ;
    ?Author (xsd:string) ;
    ?Timestamp (xsd:dateTime) ;
    ?cursor ("next_cursor" xsd:string) .
}

Offset-Based Example

The SERVICE clause below shows an example that uses the s:offset property to configure offset-based paging.

SERVICE <http://cambridgesemantics.com/services/DataToolkit>
{
  ?api a s:HttpSource ;
    s:url "http://slack.com/api/messages" ;
    s:parameter [ s:name "offset" ; s:value ?offset ] ;
    s:parameter [ s:name "limit" ; s:value ?limit ] ;
    s:limit 1000 ;
    s:paging [
      s:offset ?offset ;
      s:limit ?limit ;
      s:size 100 ;
    ] ;
    ?Message (xsd:string) ;
    ?Author (xsd:string) ;
    ?Timestamp (xsd:dateTime) .
}