BLACK OPS SOLUTIONS · IT Graduate IT Interview PackAU · 2026

AI / ML Engineer · graduate level · Australia

Graduate AI Engineer

Builds language-model features that have to work on real, messy inputs.

Job description · fictional employer

Graduate AI Engineer

Corella AI

Location
Sydney - hybrid, 3 days in office
Employment type
Full-time, permanent - 12-month graduate program
Salary
AUD $92,000 base + 12% superannuation + equity participation
Reports to
Head of Applied AI
Intake
January 2027 - applications close 31 August 2026

About us

Corella AI builds document intelligence for insurance and legal firms. Our customers send us claim files, policy schedules and discovery bundles - scanned, rotated, handwritten in the margins, forty years old - and expect structured answers with a citation for every one. We are 34 people, eleven of them engineers.

The team you would join

Applied AI is five engineers who sit between research and product. We do not train foundation models. We build the systems around them: retrieval, prompting, tool use, evaluation, guardrails and the unglamorous data plumbing that decides whether any of it works.

What you will do

  • Build and improve retrieval-augmented generation pipelines - chunking, embedding, retrieval, reranking, prompt assembly
  • Write evaluations before you write features. Every change ships with a measurement
  • Build labelled datasets from real customer documents, including doing some of the labelling yourself
  • Run structured error analysis: read failures one at a time and categorise them, do not skim aggregate scores
  • Serve models and pipelines behind FastAPI services, with sensible timeouts, retries and cost controls
  • Work directly with two named customers on accuracy problems, including reading their documents
  • Contribute to prompt and model version control - we treat prompts as code, reviewed and tested
  • Investigate a new technique each quarter and present whether it is worth adopting, with evidence

What we are looking for

  • A completed or in-progress bachelor or masters degree in computer science, data science, mathematics, statistics, engineering or a related quantitative discipline
  • Strong Python - you can write a class, handle exceptions properly, and use a virtual environment without help
  • Working understanding of core machine learning: train and test splits, overfitting, evaluation metrics, why accuracy is a poor metric on imbalanced data
  • Hands-on experience building something with a large language model API - a project, a hackathon, a thesis chapter
  • The instinct to measure rather than eyeball. If you cannot tell us how you would know your change helped, this role will be hard
  • Ability to read a paper or a model card and extract what is actually claimed
  • Full Australian working rights

Nice to have

  • PyTorch or JAX beyond a tutorial
  • Any experience with vector search, embeddings or reranking models
  • Exposure to evaluation frameworks, or to writing your own
  • Data engineering fundamentals - SQL, pandas, working with awkward file formats
  • Any experience with document processing, OCR or information extraction
  • A public writeup of something you tried that did not work

Our stack

Python 3.12PyTorchHugging Face TransformersFastAPIPostgreSQL + pgvectorAnthropic and OpenAI APIsAWS BedrockMLflowDockerPrefectWeights and Biases

What the program gives you

  • A first project with a real customer and a real accuracy target in week three
  • One day a fortnight for research reading, with a paper discussion the team actually attends
  • Compute budget for experiments, and no requirement to justify a failed one
  • Mentoring from an engineer who has shipped model-backed features to production
  • Conference or workshop attendance annually, and support to publish or present internal work

How the process runs

  1. 1

    Application

    CV plus a link to something you built with a model, and two paragraphs on what did not work about it.

  2. 2

    Take-home

    Four hours, paid. A small extraction task on messy documents, with a held-out test set you do not see.

  3. 3

    Technical interview

    75 minutes - walkthrough of the take-home, ML fundamentals, and how you would improve your own result.

  4. 4

    Applied AI interview

    60 minutes - retrieval, evaluation design, failure analysis, cost and latency trade-offs.

  5. 5

    Team and values interview

    45 minutes - communication, intellectual honesty, working with non-technical customers.

  6. 6

    Offer

    Within a week of the final stage.

We are more interested in how you think about being wrong than in how many models you can name. Candidates who bring a project where they measured something, found it disappointing, and worked out why do consistently well here. Reasonable adjustments are available at any stage.

Interview questions · 21 questions with model answers

Graduate AI Engineer

Answers are hidden by default so you can attempt each one first.

Motivation and behavioural

Intellectual honesty is the trait being tested. AI work punishes people who cannot say 'that did not work'.

  1. Tell me about something you built with a model that did not work as well as you hoped. What did you learn?

    Show what a strong answer coversHide answer

    A strong answer

    • Has a concrete failure and can describe it precisely
    • Diagnosed it rather than abandoning it - data quality, leakage, a bad metric, an unrepresentative test set
    • Distinguishes 'the model was wrong' from 'I measured the wrong thing'
    • Is comfortable being unimpressive about their own work

    Red flagEvery project in their history worked on the first attempt.

  2. How do you keep up with the field without drowning in it?

    Show what a strong answer coversHide answer

    A strong answer

    • Has a filter - a few sources, a rule for what to read deeply
    • Distinguishes benchmark news from things that change practice
    • Can name a technique they decided not to adopt and say why
    • Has actually run something rather than only read about it

    Red flagLists newsletters and cannot say what they changed as a result.

  3. A customer insists the output is wrong. You look and think the output is right. How do you handle it?

    Show what a strong answer coversHide answer

    A strong answer

    • Assumes the disagreement is real information, not a customer error
    • Gets the specific example rather than arguing about the general case
    • Recognises the definition of correct may differ from theirs
    • Turns the disagreement into a test case either way

    Red flagWants to explain to the customer why they are mistaken.

  4. How would you explain to a claims manager why the system sometimes gets things wrong?

    Show what a strong answer coversHide answer

    A strong answer

    • Plain language, no jargon and no hand-waving about neural networks
    • Honest about probabilistic behaviour without being fatalistic
    • Explains what the guardrails and citations are for
    • Gives them something actionable - what to check, how to report it

    Red flagEither promises it will be fixed, or hides behind complexity.

  5. What worries you about the work this team does?

    Show what a strong answer coversHide answer

    A strong answer

    • Has actually thought about failure modes on legal and insurance documents
    • Mentions hallucinated citations, silent errors, over-trust, privacy of customer documents
    • Is thoughtful rather than performatively worried
    • Connects it to something they would do about it

    Red flagHas no concerns at all, or only rehearsed talking points.

Machine learning fundamentals

Do not skip these because the role is LLM-focused. Grads who cannot reason about evaluation build things that look right and are not.

  1. You have a classifier that is 97% accurate. Why might I not be happy about that?

    Show what a strong answer coversHide answer

    A strong answer

    • Immediately asks about class balance
    • Knows that a 97%-negative dataset makes 'always no' a 97% model
    • Reaches for precision, recall, F1, or a confusion matrix
    • Asks which error is more expensive in this business context

    Red flagTakes 97% as good news.

  2. Explain precision and recall to me, and tell me which one matters more for flagging fraudulent insurance claims.

    Show what a strong answer coversHide answer

    A strong answer

    • Correct definitions, ideally with the confusion matrix in mind
    • Recognises the trade-off is a business decision, not a technical one
    • Reasons about the cost of a missed fraud versus the cost of accusing a legitimate customer
    • Mentions that a flag feeding a human reviewer changes the answer

    Red flagMixes them up and cannot recover when you give a worked example.

  3. What is overfitting, and how would you detect it in a model someone else trained?

    Show what a strong answer coversHide answer

    A strong answer

    • Learning the training set rather than the pattern
    • Compares training and held-out performance
    • Checks how the split was made - random splits leak when documents or customers repeat
    • Mentions regularisation, more data, simpler models or early stopping as responses

    Red flagDefines it correctly but has never checked for it.

  4. What is data leakage, and can you give me an example that would be easy to miss?

    Show what a strong answer coversHide answer

    A strong answer

    • Information in training that would not be available at prediction time
    • Good examples: a field populated after the outcome, duplicate documents split across train and test, normalising before splitting
    • Knows leakage shows up as suspiciously good results
    • Would check by looking at what the model relies on

    Red flagHas not encountered the concept and does not ask.

  5. What is an embedding, and why does cosine similarity between two embeddings mean anything?

    Show what a strong answer coversHide answer

    A strong answer

    • A dense vector where geometric closeness approximates semantic closeness
    • Knows similarity is defined by what the encoder was trained on, not by universal truth
    • Mentions that domain mismatch degrades it - legal documents against a general-purpose encoder
    • Bonus: knows similarity is not the same as relevance to a question

    Red flagTreats embeddings as magic that always works.

  6. Why is a random train-test split sometimes the wrong thing to do?

    Show what a strong answer coversHide answer

    A strong answer

    • Time series - training on the future to predict the past
    • Grouped data - the same customer, document or patient in both sets
    • Suggests time-based or group-based splitting
    • Connects it to the model looking better offline than in production

    Red flagBelieves random is always correct.

Applied LLM engineering

This is the day job. Depth here separates candidates who have shipped from those who have prompted.

  1. Walk me through a RAG pipeline. Where does it usually break?

    Show what a strong answer coversHide answer

    A strong answer

    • Ingest, chunk, embed, index, retrieve, rerank, assemble prompt, generate, cite
    • Says retrieval is usually the problem, not generation
    • Names real failure points: bad chunking splitting a table, the answer spanning two chunks, a query that does not lexically match the source
    • Mentions hybrid search or reranking as responses

    Red flagDescribes it as 'you give the model your documents'.

  2. How would you evaluate a system that answers questions about insurance policies, when there is no single right answer?

    Show what a strong answer coversHide answer

    A strong answer

    • Builds a labelled set from real questions, even a small one
    • Separates retrieval evaluation from answer evaluation
    • Uses graded criteria - is it supported by the cited text, is it complete, does it hedge appropriately
    • Knows model-as-judge needs its own validation against human labels
    • Insists on holding some examples out

    Red flagWould 'just try it and see if the answers look good'.

  3. What is hallucination, and what actually reduces it in a production system?

    Show what a strong answer coversHide answer

    A strong answer

    • Fluent output unsupported by any source
    • Grounding in retrieved text with enforced citations
    • Allowing and rewarding 'I do not know' as an output
    • Verification passes, constrained output formats, checking cited spans actually exist in the source
    • Knows prompting alone is a weak control

    Red flagSays 'better prompts' and stops.

  4. Your accuracy is acceptable but each request costs $0.40 and takes 11 seconds. What do you do?

    Show what a strong answer coversHide answer

    A strong answer

    • Measures where the time and money go before optimising
    • Considers a smaller model for easy cases and routing hard ones up
    • Caching, shorter context, fewer retrieved chunks, parallel calls, streaming for perceived latency
    • Asks what the user actually needs - 11 seconds may be fine for a batch job and fatal in a chat

    Red flagImmediately suggests fine-tuning as the first move.

  5. When would you fine-tune rather than improve prompting and retrieval?

    Show what a strong answer coversHide answer

    A strong answer

    • When the failure is format, style or a consistent domain behaviour rather than missing knowledge
    • Knows fine-tuning does not reliably add facts
    • Weighs the cost: labelled data, retraining on model upgrades, evaluation burden
    • Would exhaust retrieval and prompting first

    Red flagTreats fine-tuning as the default answer to any quality problem.

  6. How would you stop a document-processing agent from doing something it should not?

    Show what a strong answer coversHide answer

    A strong answer

    • Constrain the tools available rather than asking nicely in the prompt
    • Validate outputs against a schema and reject rather than repair silently
    • Human approval on irreversible actions
    • Limits on iterations, spend and time; logging every tool call
    • Knows prompt injection from document content is a real threat when documents are untrusted

    Red flagRelies entirely on instructions in the system prompt.

Practical and take-home walkthrough

Anchored on the paid take-home. Let them lead, then push one level past their comfort.

  1. Talk me through your take-home. What did you measure, and what was your baseline?

    Show what a strong answer coversHide answer

    A strong answer

    • Established a baseline before optimising - even a trivial one
    • Can state their metric and defend the choice
    • Knows which of their changes helped and which did not
    • Says what they ran out of time to do

    Red flagReports one final number with no baseline and no error analysis.

  2. Your extraction is 82% accurate. I need 95%. Where do you start?

    Show what a strong answer coversHide answer

    A strong answer

    • Looks at the 18% first and categorises the failures
    • Distinguishes systematic errors from long-tail ones
    • Checks whether some labels are simply wrong
    • Asks whether 95% overall is the real requirement, or 95% on the fields that matter
    • Willing to say the target may not be reachable with the current approach

    Red flagStarts trying different models or prompts at random.

  3. The customer sends 300-page scanned PDFs, some rotated, some handwritten. How do you get from that to something a model can use?

    Show what a strong answer coversHide answer

    A strong answer

    • Thinks about the pipeline before the model - OCR quality, orientation detection, layout parsing, tables
    • Knows garbage extraction caps everything downstream
    • Would sample and measure OCR quality rather than assume it
    • Considers handling handwriting separately, or routing it to a human

    Red flagAssumes the text extraction step is solved.

  4. How would you know, three months after launch, that quality had quietly degraded?

    Show what a strong answer coversHide answer

    A strong answer

    • Monitoring in production, not just an offline test set
    • Tracks proxy signals: user corrections, escalations, retrieval scores, refusal rates, input distribution shift
    • Wants a periodically re-labelled sample of live traffic
    • Mentions that a model version change upstream can move behaviour overnight

    Red flagAssumes offline evaluation before launch is sufficient.

Questions to ask them

Bring three. Interviewers remember the candidate who asked something they had to think about.

  • What does your evaluation set look like, and who built it?
  • When a customer reports a bad output, what path does that take to becoming a test case?
  • How do you decide between improving retrieval and improving the prompt? Is that instinct or measurement here?
  • How much of my time would be model work versus data plumbing? I would rather know honestly.
  • What has the team tried in the last six months that you decided not to keep?

Example CV · fictional candidate

Wei Zhang

Written to the job description on the previous tab. Notes on the right explain each choice.

Wei Zhang

Graduate AI / Machine Learning Engineer

Sydney NSW · 0400 000 000 · w.zhang@example.com · github.com/weizhang-ml · weizhang.example.com (writeups)

Professional summary

Advanced computing honours graduate focused on applied language-model systems. Honours thesis on retrieval quality in domain-specific question answering, plus a summer building an evaluation harness that a commercial team still uses. I measure before and after every change, and I write up the things that did not work.

Technical skills
Languages
Python (primary), SQL, C++ (coursework), JavaScript (basic)
ML and DL
PyTorch, scikit-learn, Hugging Face Transformers, sentence-transformers
LLM systems
RAG pipelines, pgvector, FAISS, reranking, structured output, evaluation harnesses, prompt versioning
Data
pandas, NumPy, PostgreSQL, Polars (basic), PDF and OCR tooling
MLOps
MLflow, Weights and Biases, Docker, FastAPI, GitHub Actions
Maths
Linear algebra, probability and statistics, optimisation (university level)
Education
Bachelor of Advanced Computing (Honours Class I), majoring in Machine Learning and Data Science
Feb 2023 - Dec 2026

University of Sydney

  • WAM 82. Honours thesis mark 89
  • Thesis: retrieval quality as the limiting factor in domain-specific question answering. Built a 480-question labelled evaluation set over Australian tenancy legislation and showed reranking recovered 21 points of answer accuracy where prompt changes recovered 4
  • Relevant units: Statistical Machine Learning (HD), Deep Learning (HD), Natural Language Processing (HD), Optimisation (D), Database Systems (D)
Experience
Machine Learning Intern
Nov 2025 - Feb 2026 (14 weeks)

Wrenfield Analytics, Sydney

  • Built the team's first automated evaluation harness for an internal document classifier: 340 labelled examples, per-class metrics, regression checks in CI. Still in use
  • Ran error analysis on 200 misclassifications and found 38 were mislabelled in the ground truth, which changed the reported baseline by 4 points
  • Reduced average inference cost per document by 46% by routing short documents to a smaller model and only escalating on low confidence
  • Wrote the internal note explaining why a proposed fine-tune was not worth doing, which the team accepted
Undergraduate Research Assistant (casual, 10 hrs/week)
Mar 2025 - Nov 2025

USyd School of Computer Science

  • Prepared and cleaned a 1.2 million document corpus for a supervisor's NLP project, including deduplication that removed 14% of near-duplicates
  • Reproduced results from two published papers and documented where the reported numbers could not be reproduced
Mathematics Tutor (casual)
Feb 2023 - Dec 2025

Private and school-based tutoring

  • Taught HSC Extension 1 and 2 mathematics to 12 students across three years
  • Direct practice at explaining technical ideas to people who do not yet have the vocabulary
Projects
Tenancy QA - grounded question answering over legislation
Python, pgvector, sentence-transformers, cross-encoder reranking, FastAPI
  • Hybrid retrieval with reranking over 2,400 sections of state tenancy legislation, every answer citing a section number
  • Published the 480-question evaluation set and a writeup of three approaches that made results worse
  • github.com/weizhang-ml/tenancy-qa
Receipt extraction from photographs
PyTorch, OCR, layout-aware extraction
  • End-to-end extraction of merchant, date, total and line items from phone photos, tested on 600 self-collected receipts
  • Documented that accuracy fell from 91% to 63% on receipts printed on thermal paper more than a year old, and why
Writeups
Technical writing
  • Nine posts on evaluation design and retrieval failures, including 'Three RAG improvements that made my system worse'
Leadership and activities
  • Co-organiser, USyd Machine Learning Society reading group, 2025 - 2026. Ran fortnightly paper discussions with 20 to 30 attendees
  • Kaggle - top 8% in a document classification competition, 2025. Writeup published
Certifications
  • DeepLearning.AI Natural Language Processing Specialisation, 2024
Referees

Available on request.