Design data processing systems

​

This standard is about designing data processing systems.

Data engineers prepare data for analytical or operational uses. They are typically responsible for designing and building data pipelines to bring together information from different source systems. They integrate, consolidate, cleanse and structure data to make it easily accessible and in usable format. Processed data can then be used by business executives, data analysts and other end users to inform organisational processes and decision making.

Designing data processing systems involves selecting data storage systems, designing data pipelines, and implementing design approaches for security, scalability, efficiency, flexibility and portability. It also includes testing and documenting data processing design solutions.

This standard is for those who need to design data processing systems as part of their duties.

​

1. Review requirements to plan data processing system specifications
2. Translate business requirements into accurate data solution designs and roadmaps
3. Produce and interpret data models to determine the relationships and data flows required
4. Identify opportunities to reuse existing data flows to improve efficiency

5. Reverse engineer data models from live systems to model data structures 

6. Design data extraction and manipulation routines to process data into useable information

7. Identify data flows and data lineage, to show which parts of the organisation generate and utilise data, and how data moves through the organisation

8. Develop designs of data models, data pipelines and data services in line with organisational requirements

9. Select storage technologies for data system designs in line with organisational requirements

10. Verify the organisational procedures for data security and compliance to inform data system designs

11. Review and apply organisational standards for scalability, efficiency, reliability, availability, flexibility and portability to inform data system designs
12. Document data system design solutions in line with organisational standards

13. Automate manual data flows to enable scaling and repeatable use

14. Test data system solutions in line with system requirements

15. Document data processing designs to inform system developers
16. Provide technical advice and guidance to support system developers and end users

​

1. That organisational data is an asset with unique properties that influence its management
2. The data management practices used by an organisation to maintain high quality data
3. The types of data structures and architectures used in organisations
4. The main principles of data access, privacy and security and how to apply these to data design

5. How to develop, interpret and compare data models

6. Where to use different types of data models

7. The industry standard tools for data design and how to apply them
8. Cloud data platforms and data storage technologies
9. How to map storage systems to data processing requirements 
10. Industry standard data modelling patterns and standards
11. Industry standard systems used for on-premises and cloud-based data storage and their implications on data privacy
12. How to keep updated on data technologies and platforms

13. Corporate, industry and professional data standards

14. Industry and organisational standards for data management including scalability, efficiency, reliability, availability, flexibility, portability and quality

15. How to incorporate security into data processing design

16. The data policy, legislative, regulatory and operational constraints which exist within the organisation

17. The industry standard technologies and design principles involved in batch and streaming data processing

18. The steps involved in data staging and how to apply them

19. How to document data designs