Build and implement data processing systems

​

This standard is about building and implementing data processing systems.

Data engineers prepare data for analytical or operational uses. They are typically responsible for designing and building data pipelines to bring together information from different source systems. They integrate, consolidate, cleanse and structure data to make it easily accessible and in usable format. Processed data can then be used by business executives, data analysts and other end users to inform organisational processes and decision making.

Building and implementing data processing systems involves creating data storage to link data systems, and creating and deploying data pipelines and data processing infrastructure. This also includes implementing batch and stream processing solutions, building and running tests to confirm required operation and documenting implementations.

This standard is for those who need to build and implement data processing systems as part of their duties.

1. Review data processing system designs to plan system development
2. Plan data processing development activities in line with organisational procedures
3. Build and debug data pipelines to process and deliver data to destination systems
4. Integrate data processing systems with data storage systems and business processes

5. Implement data security controls into data processing systems in line with organisational procedures

6. Implement data flows and transformations to deliver data to users

7. Configure data pipelines in line with organisational requirements

8. Automate data processing to provide adaptive operation
9. Build and run test suites to validate data pipeline functionality in line with organisational requirements
10. Deploy data solutions into production environments to make them operational
11. Manage commercial relationships with data engineering partners in line with organisational procedures
12. Document data pipeline solutions in line with organisational standards

​