United Kingdom Job Openings
Adroit People Ltd.
AWS Data Engineer
Telford
FULL TIME
November 8, 2024
• Design, develop, and maintain real-time data streaming pipelines using AWS Kinesis for processing and analyzing high-velocity data streams.
- Collaborate with stakeholders to gather requirements and design efficient data streaming architectures for real-time analytics, monitoring, and alerting.
- Build and optimize Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics to process real-time data from various sources.
- Develop and implement ETL processes to transform, enrich, and aggregate streaming data for downstream consumption in data lakes or data warehouses such as AWS S3, Redshift, and Dynamo DB.
- Integrate Kinesis with other AWS services like Lambda, S3, and EMR for real-time data ingestion, processing, and storage.
- Optimize streaming applications for performance and scalability, ensuring efficient resource usage and minimal latency.
- Develop and maintain robust error handling, data validation, and retry mechanisms within Kinesis pipelines to ensure data consistency and reliability.
- Implement security best practices for data streaming, including encryption in transit and at rest, access control using IAM roles and policies, and data retention policies.
- Monitor and troubleshoot AWS Kinesis streams using Cloud Watch metrics, alarms, and logs, ensuring high availability and minimal downtime.
- Collaborate with Dev Ops teams to automate the deployment and scaling of streaming pipelines using AWS Cloud Formation, Lambda, and CI/CD tools.
- Work closely with data analysts and data scientists to ensure real-time data is processed and delivered for immediate insights and reporting.
- Implement data partitioning, sharding, and key management strategies to handle high-throughput data streams and ensure scalability.
- Optimize the cost of real-time data processing by tuning the configurations of Kinesis and associated AWS services like Lambda and Firehose.
- Stay current with AWS updates and best practices for streaming data, recommending improvements to the architecture as new features and services become available.
- Provide guidance and mentorship to junior engineers in real-time data processing, ensuring best practices in streaming technologies and AWS cloud services.
- Create and maintain detailed technical documentation for data pipelines, including data flow diagrams, stream processing logic, and operational guidelines.
Experience:
- Kinesis Data Streams: 8 years (preferred)
- AWS: 8 years (preferred)
- United Kingdom (preferred)
New Job Alerts
Kodiform Therapeutics Limited
Scientist (Cell & Molecular Biology)
London
FULL TIME
November 18, 2024
View Job Description
Home Office
Authorising Officer
Birmingham
FULL TIME & PART TIME
November 18, 2024
View Job DescriptionFrederic Robinson Brewery
Housekeepers
Ilkley
FULL TIME & PART TIME
November 18, 2024
View Job Description
Looking for similar job?
Pinnacle Group
Junior Data Analyst - Holborn, England
Holborn
FULL TIME
August 22, 2024
View Job Description
RWS Group
Data Annotator for AI Models | English | Remote, Part Time, Work from Home
PART TIME & CONTRACT
August 1, 2024
View Job Description
FDM Group (Graduate)
Data & Analytics Graduate Programme - Dublin
FULL TIME
August 14, 2024
View Job Description
Alibaba
Alibaba Cloud-Data Center Asset Manager (EMEA)-London, United Kingdom
London
August 6, 2024
View Job Descriptione-Careers Limited
Trainee Data Analyst
Birmingham
FULL TIME & PART TIME
August 25, 2024
View Job DescriptionSee What’s New: Adroit People Ltd. Job Opportunities
Adroit People Ltd.
Software Asset Management Consultant
London
November 18, 2024
View Job Description
New Job Alerts
Kodiform Therapeutics Limited
Scientist (Cell & Molecular Biology)
London
FULL TIME
November 18, 2024
View Job Description
Home Office
Authorising Officer
Birmingham
FULL TIME & PART TIME
November 18, 2024
View Job DescriptionFrederic Robinson Brewery
Housekeepers
Ilkley
FULL TIME & PART TIME
November 18, 2024
View Job Description