Data Processing + Feed Integration Software Engineer

4 days ago

Wellington Central, Wellington, New Zealand Lumini Works Ltd Full time NZ$80,000 - NZ$120,000 per year
About the Role

We're seeking a highly skilled Data Processing & Feed Integration Engineer to join our team and take ownership of our sophisticated document processing and data ingestion platform. This role focuses on building robust data collection systems, processing diverse content sources, and ensuring seamless integration of new data feeds into our AI-powered document analysis platform.

Key Responsibilities

Data Collection & Processing

  • Design and implement automated data collection systems from various web sources, including RSS feeds, social media, government websites, and parliamentary systems
  • Develop robust web content extraction pipelines using modern tools and techniques to handle diverse document formats (PDF, HTML, DOCX, PPTX, VIDEO formats)
  • Build intelligent content processing workflows that automatically classify, chunk, and prepare documents for AI analysis
  • Implement sophisticated retry mechanisms and error handling for reliable data collection from external sources
  • Create scalable document processing pipelines that handle large volumes of content efficiently

Feed Integration & Onboarding

  • Onboard new data sources and feeds by analysing source structures, authentication requirements, and content patterns
  • Develop custom parsers and processors for specialised content types, including parliamentary Hansard debates, government consultations, and policy documents
  • Implement intelligent cookie management and session handling for authenticated data sources
  • Design flexible feed configuration systems that support various monitoring intervals, content filtering, and processing rules
  • Create comprehensive testing frameworks for new feed integrations and data processing workflows

Technical Skills Required

Core Technologies

  • TypeScript/JavaScript - Advanced proficiency in modern JavaScript development
  • Supabase & PostgreSQL - Extensive experience with database design, migrations, and performance optimisation
  • Vector Databases - Experience with pgvector and semantic search implementations
  • Serverless Architecture - Deep understanding of edge functions and cloud computing patterns

Data Processing & Web Technologies

  • Web Data Extraction & Collection - Advanced skills in building robust data gathering systems
  • RSS/XML Processing - Experience in parsing and processing various feed formats
  • Document Processing - Expertise in handling PDF, HTML, DOCX, and other document formats
  • HTTP/API Integration - Strong skills in building reliable external service integrations
  • Cookie Management & Authentication - Experience with session handling and authenticated data access

Error Handling & Resilience - Strong skills in building fault-tolerant systems

Experience Requirements

  • 2+ years of experience in data processing, web data collection, or content management systems
  • 2+ years of experience with modern JavaScript/TypeScript development
  • 2+ years of experience with PostgreSQL and database design
  • Proven track record of building scalable data processing pipelines
  • Experience with government or policy-related data sources (highly desirable)

Personal Attributes

  • Problem-solving mindset with the ability to tackle complex data integration challenges
  • Attention to detail when working with diverse data formats and edge cases
  • Proactive approach to system monitoring and performance optimisation
  • Adaptability to work with evolving data sources and processing requirements

  • Data Engineer

    4 days ago

    Wellington Central, Wellington, New Zealand Office of the Auditor-General Full time NZ$80,000 - NZ$120,000 per year

    Data Engineer (12-month fixed term)The Office of the Auditor-General | Te Mana Arotake aims to improve trust and confidence in public services by giving New Zealanders an independent view of government spending and performance. As the statutory auditor of all public organisations, from councils to schools, the Auditor-General's work plays a major part in...

  • Principal / Senior Data Engineer (Data Platforms)

    4 days ago

    Wellington, Wellington, New Zealand Simple Machines Full time NZ$120,000 - NZ$180,000 per year

    Principal / Senior Data Engineer (Data Platforms)Wellington, New ZealandSimple Machines. Data Engineered to LifeSimple Machines is a leading independent boutique technology firm with a global presence, including teams in London, Sydney, San Francisco, and New Zealand. We specialise in creating technology solutions at the intersection of data, AI, machine...

  • Principal / Senior Data Engineer (Data Platforms)

    3 hours ago

    Wellington, Wellington, New Zealand Simple Machines Full time US$80,000 - US$160,000 per year

    **Principal / Senior Data Engineer (Data Platforms)Wellington, New Zealand**Simple Machines. Data Engineered to LifeSimple Machines is a leading independent boutique technology firm with a global presence, including teams in London, Sydney, San Francisco, and New Zealand. We specialise in creating technology solutions at the intersection of data, AI, machine...

  • Software Engineer

    3 hours ago

    Wellington, Wellington, New Zealand Trimble Inc. Full time NZ$85,000 - NZ$125,000 per year

    Your Title: Software EngineerJob Location: Wellington, NZOur Department: Trimble Field Systems - CloudAre you a Software Engineer who is excited about working in Backend Development, DevOps, and Machine Learning, in an industry-leading global tech company?What You Will DoIf you are a Software Engineer with a passion for Backend Development, DevOps, and...

  • MLOps Engineer

    2 days ago

    Wellington Central, Wellington, New Zealand Docuvera Software Corporation Full time NZ$120,000 - NZ$180,000 per year

    At Docuvera, we're transforming how the Life Sciences industry creates, manages, and reuses content. As a global SaaS leader in structured content authoring, our mission is to drastically improve time to market, productivity, patient safety, and patient outcomes. Our platform empowers pharmaceutical companies worldwide to drive digital transformation with...

  • Data and Application Infrastructure Engineer

    2 days ago

    Wellington, Wellington, New Zealand DOWNER NZ Full time

    Join Downer Auckland as a Data & Application Infrastructure Engineer and help drive innovation, reliability, and performance across our digital platformsAt Downer we plan, create and sustain. Downer is the leading provider of integrated services in Australia and New Zealand. Through trusted relationships and world leading insight we work closely with our...

  • Associate Software Engineer

    4 days ago

    Wellington Central, Wellington, New Zealand Transpower Full time NZ$90,000 - NZ$120,000 per year

    Transpower is the owner and operator of the National Grid. At the heart of the electricity sector, we transport electricity via high voltage transmission lines from where it is generated to large electricity users and the distribution businesses that deliver it to homes and businesses nationwide.Our core purpose is to empower the energy future for New...

  • Senior Process Engineer

    4 days ago

    Wellington Central, Wellington, New Zealand Stantec New Zealand Full time NZ$90,000 - NZ$120,000 per year

    Senior Process Engineer – WellingtonCommunities are fundamental. Whether around the corner or across the globe, they provide a foundation, a sense of place and of belonging. That's why at Stantec, we always design with community in mind.We care about the communities we serve—because they're our communities too. We're designers, engineers, scientists, and...

  • Lead Data Engineer

    4 days ago

    Wellington, Wellington, New Zealand MAS - Medical Assurance Society Full time NZ$120,000 - NZ$180,000 per year

    We're seeking a Lead Data Engineer to join MAS and contribute to our future success as we continue to transform our data and analytics practice and build out our Modern Data Platform.The Lead Data Engineer is an integral part of the Data & Analytics team, and is responsible for the design, management & implementation of complex data flows to connect MAS's...

  • Data Engineer Data Solutions

    4 days ago

    Wellington Central, Wellington, New Zealand Oranga Tamariki - Ministry for Children Full time NZ$125,926 - NZ$139,459 per year

    1x fulltime Fixed Term till June 2026 availableRole is based at National Office, Te Whanganui a Tara/WellingtonWe're offering a competitive salary starting from $125,926 to $139,459 depending on experienceMō mātou | About usOranga Tamariki—Ministry for Children has a commitment to putting tamariki and rangatahi at the heart of what we do. In our work, we...