The Macrobond FTP/SFTP data feed

Introduction

The Macrobond FTP/SFTP data feed allows automated integration of our economic and financial data with your in-house backend systems. Below we describe basic feed concepts and the various types of feed service we offer.

The Macrobond FTP/SFTP data feed is a separately licensed solution.

Working with FTP data feed

Data Delivery

For maximum flexibility, updates are provided in XML format. This allows for easy integration with your data processing system(s). Updates are provided on Macrobond servers and are reachable via FTP, SFTP or HTTPS protocols. Update documents are generated every time new data is integrated into the Macrobond database, or when existing data is revised.

You will receive a dump of the full history once a week, covering the entire set of subscriptions.

You will receive list of all the subscribed series with information about their status every 24h.

Data Set Selection

Depending on your requirements, you can declare the name of the time series you wish to retrieve within the Macrobond’s data feed online portal.

Content

Content of the Data Feed Folder

All feed files are stored in the root folder of your FTP account.

Update documents

Update documents are generated every time new data is integrated into the Macrobond database and are named using the following standard: {number}.xml e.g.:

-rw-r--r-- 1 1000 1000 29488 Dec 16 11:06 1.xml
-rw-r--r-- 1 1000 1000 5145 Dec 20 17:01 10.xml
-rw-r--r-- 1 1000 1000 29508 Dec 21 11:04 11.xml
-rw-r--r-- 1 1000 1000 5159 Dec 21 17:00 12.xml
-rw-r--r-- 1 1000 1000 29514 Dec 22 11:05 13.xml
-rw-r--r-- 1 1000 1000 5173 Dec 22 13:58 14.xml
-rw-r--r-- 1 1000 1000 5173 Dec 22 17:00 15.xml
-rw-r--r-- 1 1000 1000 29521 Dec 23 11:04 16.xml
-rw-r--r-- 1 1000 1000 5187 Dec 23 17:00 17.xml
-rw-r--r-- 1 1000 1000 5201 Dec 24 08:01 18.xml
-rw-r--r-- 1 1000 1000 5201 Dec 24 17:00 19.xml
-rw-r--r-- 1 1000 1000 5118 Dec 16 17:00 2.xml
-rw-r--r-- 1 1000 1000 29529 Dec 27 11:07 20.xml
-rw-r--r-- 1 1000 1000 5215 Dec 27 17:00 21.xml
-rw-r--r-- 1 1000 1000 29536 Dec 28 11:05 22.xml
-rw-r--r-- 1 1000 1000 5229 Dec 28 17:01 23.xml
-rw-r--r-- 1 1000 1000 29543 Dec 29 11:06 24.xml
-rw-r--r-- 1 1000 1000 29495 Dec 17 11:07 3.xml
-rw-r--r-- 1 1000 1000 5131 Dec 17 17:01 4.xml
-rw-r--r-- 1 1000 1000 5145 Dec 20 10:32 5.xml
-rw-r--r-- 1 1000 1000 5145 Dec 20 10:47 6.xml
-rw-r--r-- 1 1000 1000 29502 Dec 20 11:07 7.xml
-rw-r--r-- 1 1000 1000 5145 Dec 20 11:21 8.xml
-rw-r--r-- 1 1000 1000 5145 Dec 20 17:00 9.xml

Data contained in these files should be integrated into your database systems following the natural order {number} – 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 etc.

Full data dump

A full dump of documents is generated once a week using the following naming convention: full_dump_{date}_{time}s_n_of_m.xml.gz. For instance:

-rw-r--r-- 1 feed feed 14564466 Dec 2 03:10 full_dump_20211202_031014s_1_of_8.xml.gz
-rw-r--r-- 1 feed feed 13835432 Dec 2 03:10 full_dump_20211202_031014s_2_of_8.xml.gz
-rw-r--r-- 1 feed feed 13669402 Dec 2 03:10 full_dump_20211202_031014s_3_of_8.xml.gz
-rw-r--r-- 1 feed feed 14003385 Dec 2 03:10 full_dump_20211202_031014s_4_of_8.xml.gz
-rw-r--r-- 1 feed feed 13446979 Dec 2 03:10 full_dump_20211202_031014s_5_of_8.xml.gz
-rw-r--r-- 1 feed feed 14194428 Dec 2 03:10 full_dump_20211202_031014s_6_of_8.xml.gz
-rw-r--r-- 1 feed feed 14071453 Dec 2 03:10 full_dump_20211202_031014s_7_of_8.xml.gz
-rw-r--r-- 1 feed feed 14222872 Dec 2 03:10 full_dump_20211202_031014s_8_of_8.xml.gz

Currently, as above, 8 files form a full dump. However, it is possible that in the future this number may change.

Files that are more than 6 days old are automatically removed from the client’s data feed folder.

Content of the Data Feed Files

Update documents are generated each time when series to which feed clients subscribe are revised or extended.

Full data dumps - generated weekly, even when the database has not been updated - use the same xml format as the update documents.

Format

The XSD file defining the xml format produced by the feed can be found at: http://www.macrobond.com/schemas/Feed.xsd

When an xml file contains series, they will be presented in the following format:

<result>
  <feed_entry document_type="full" primname="some_code0">
    <entity primname="some_code0" type="TimeSeries" state="0" changed_observations="1" changed_meta="0">
      <meta_data>
        <!-- meta-data part -->
      </meta_data>
      <observation_data>
        <!-- series observation part -->
      </observation_data >
  </entity>
</feed_entry>
<feed_entry document_type="full" primname="some_code1" changed_observations="1" changed_meta="1">
  <entity primname="some_code1" type="TimeSeries" state="0">
    <meta_data>
       <!-- meta-data part -->
    </meta_data>
    <observation_data>
       <!-- series observation part -->
    </observation_data>
   </entity>
 </feed_entry>
</result>

Each time series provided in the xml document is enclosed in the feed_entry tag and should be identified by primname attribute.

Each time series document, enclosed in an entity tag, contains two sections:

  • meta_data – metadata section containing attributes describing time series, information about start date, frequency and day-mask.
  • observation_data – containing values of the time series, separated by semicolon.

Series Metadata

Start date information is presented as: <attr dt="datetime" attr_id="9" name="StartDate" freetext="1">1971-01-04</attr>

· Frequency information is presented as: <attr dt="string" attr_id="4" name="Frequency" short_name="daily" value_id="4482">Daily</attr>

Possible values of the frequency field:

  • Annual
  • BiMonthly
  • Daily
  • Monthly
  • QuadMonthly
  • Quarterly
  • SemiAnnual
  • Weekly

Day Mask

Daily time series also contain day mask information indicating which day of the week is described by the given set of observations. The day mask is expressed as a decimal representation of binary numbers with the least significant bit – corresponding to Monday. The Monday-to-Friday working week is described as 31 [binary: 0001 1111 ].

Sample Day mask information: <attr dt="int32" attr_id="3" name="DayMask" freetext="1">31</attr>

Free-text attribute Scale tells order of magnitude for the values provided in the series observations section. For instance:

<attr freetext="1" name="Scale" attr_id="8" dt="int32">0</attr>: indicates that all values provided in the series observations should be multiplied by 100=1.

<attr freetext="1" name="Scale" attr_id="8" dt="int32">-3</attr>: observations should be multiplied by 10-3=0.001.

Please note that some of the attributes might have more than one value:

<attr dt="string" attr_id="66" name="Region" short_name="de" value_id="337">Germany</attr>

<attr dt="string" attr_id="66" name="Region" short_name="state_de_badwur" value_id="580833">Baden- Württemberg</attr>

Series Observations

Series observations are expressed as repeated xml tags containing date and observation value pairs. For instance:

<observation_data>
<observation date="1971-01-04">5.1643</observation>
<observation date="1971-01-05">5.1628</observation>
<observation date="1971-01-06">5.1614</observation>
<observation date="1971-01-07">5.1649</observation>
<observation date="1971-01-08">5.1631</observation>
</observation_data>

The Observations section contains only values for dates valid based on the StartDate/Frequency/DayMask combination.

Missing observations

It is possible that for some of the dates Macrobond will be missing observations. In this case the value of the observation tag will contain:

  • String NaN- if it is a missing value
  • The letter s- if it is known that there is no data for a given date (because of a holiday)

For instance:

<entity primname="sek" type="TimeSeries" state="0" changed_observations="1" changed_meta="0">
  <observation_data>
    <observation date="1971-01-04">5.1643</observation>
    <observation date="1971-01-05">s</observation> <!-- there was no data/trading available for this date -->
    <observation date="1971-01-06">5.1614</observation>
    <observation date="1971-01-07">NaN</observation> <!-- macrobond has missed data update for that date -->
    <observation date="1971-01-08">NaN</observation> <!-- macrobond has missed data update for that date -->
    <observation date="1971-01-11">5.1642</observation>
  </observation_data>
  <meta_data>
    <attr dt="string" attr_id="4" name="Frequency" short_name="daily" value_id="4482">Daily</attr>
    <attr dt="datetime" attr_id="9" name="StartDate" freetext="1">1971-01-04</attr>
    <!-- other attributes -->
  <meta_data>
</entity>

Replaced Series – Aliases

It is possible that some of the series you are subscribed to will be removed from the Macrobond database and replaced with another. The replacement series contain a list, separated by commas, of the substituted series codes in the attribute called Aliases.

For instance – when the series tovvg47swdaea19tu was removed from the Macrobond database, clients who subscribed to it automatically received an update for a new series called tovvg47scaea19tu.

The new series contained:

<entity primname="tovvg47scaea19tu" type="TimeSeries" state="0" changed_observations="1" changed_meta="0">
  <observation_data>
    …
  </observation_data>
  <meta_data>
    <attr dt="string" attr_id="5" name="Aliases" freetext="1">tovvg47swdaea19tu</attr>
    …
  <meta_data>
</entity>

Generated files

Subscription List files

Subscription List files are generated every 24h – under name subscription_list_{date}_{time}.csv. The same information can also be fetched via the web management interface https://feed.macrobond.com/customer/ > Subscriptions List > Download.

Files are in CSV format, with the following columns:

  • Subscribed_code - series code as subscribed,
  • Actual_code - actual series code in Macrobond database in case subscribed_code is an alias,
  • State - current state of the series in Macrobond database, has following values:
    • 0 - series is active, receives updates,
    • 1 - series has been disabled and is not available either via data feed or the Macrobond application,
    • 4 - series has been discontinued by the source and is unlikely to receive further updates,
    • -1 - series has been deleted. If the actual_code is not empty - it’ll be provided with that code,
  • available_for_feed:
    • 0 - Macrobond is not allowed to re-distribute this series via feed
    • 1 - series is available for the feed
  • last_update_time - date and time when given series was dumped the last time, empty if it has not been dumped yet,
  • last_dump_file_num - number of the file where this series was dumped most recently, 0 if it has not been dumped yet.

Feed Statistics files

Feed Statistics files are generated every 24h – under name feed_statistics_{date}_{time}.csv. They contain column header and single line with results. Columns:

  • max_allowed_subscription_size – current upper limit of the size of subscription list,
  • currently_subscription_size – total number of series, including not-available, that are currently in the subscription list