In order to decrease the time taken to implement each data feed, we designed our own in-house Feed Management System. The feed management system contains a library of standard tasks used to perform common functions (such as obtaining data from a URL or reading content from a file).
The feed management system also allows us to deploy to the cloud, which can be used to reduce the load on a client's infrastructure and therefore increase stability. It also allows new tasks to be added and be performed together as a flow, for example:
- Obtain data from one source and convert it to another format for use by a mobile device.
- Obtain data from one source, convert it, restructure it and reorder the data for display.
- Obtain data from a file or queue, convert it, restructure it and reorder it for display.
- Aggregate data from two separate sources, merge it and then format it for use by a mobile device.
- Periodically delete unwanted files.