Remote machine learning engineer interview questions and answers

Technical questions

What you know and how you'd do the work, often with a live task or test.

1. Walk me through how you would take a trained model to production.

   Why they ask

   The core of the job. They want a complete, practical picture, not just "put it in Flask".

   How to answer

   Cover packaging and versioning (model registry), serving (API or batch job), containerisation and deployment, a safe rollout (shadow or canary), monitoring (latency, errors, input drift, outcome metrics) and a retraining plan.

   Show a sample answerHide the sample answer

   I'd register the model in MLflow with its training data version and metrics. Then wrap it in a FastAPI service, build a Docker image in CI, run tests including a few known inputs with expected outputs, and deploy to Kubernetes. I'd start in shadow mode, scoring real traffic without acting on it, compare against the current system, then roll out to 10% of traffic. Dashboards track latency, errors and feature distributions, with alerts if they drift.

2. When would you use batch inference instead of online inference?

   Why they ask

   A basic architecture choice with big cost and complexity consequences.

   How to answer

   Batch: predictions computed on a schedule and stored, fine when results aren't needed instantly (daily churn scores, weekly recommendations). Online: predictions on request with low latency, needed when the input only exists at that moment (fraud on a payment, search ranking).

   Show a sample answerHide the sample answer

   Churn scores for the customer success team can be computed nightly and written to a table, which is cheap and easy to debug. A fraud check on a card payment has to happen in real time, so that needs an online service with a tight latency budget.

3. What is training/serving skew, and how do you prevent it?

   Why they ask

   One of the most common reasons good models fail in production.

   How to answer

   It's when the features or data the model sees in production differ from training, often because they're computed by different code. Prevent it by sharing feature code between training and serving (or using a feature store), logging production inputs and comparing their distributions to training data.

   Show a sample answerHide the sample answer

   On one project, training computed "average transaction value" over 30 calendar days in SQL, but the live service used the last 30 transactions in Python. Same name, different feature. Now I keep feature logic in one shared module or a feature store, and log live inputs so I can compare them with the training set.

4. What is the difference between data drift and concept drift, and how would you monitor them?

   Why they ask

   Models decay over time. They want to know you'll notice before the business does.

   How to answer

   Data drift: the input distribution changes (new customer segments, a new app version). Concept drift: the relationship between inputs and the target changes (fraudsters change tactics). Monitor input distributions with statistics like PSI or KS tests, and track real performance once labels arrive.

   Show a sample answerHide the sample answer

   If we launch in a new state and the age and income mix of applicants changes, that's data drift. If fraudsters learn our rules and start doing smaller transactions, the same inputs now mean something different: concept drift. I'd track feature distributions daily and alert on big shifts, and measure precision and recall weekly once chargebacks come in.

5. Your GPU runs out of memory during training. What can you do?

   Why they ask

   A daily practical problem, especially when fine-tuning larger models.

   How to answer

   Reduce the batch size and use gradient accumulation to keep the effective batch size, use mixed precision (fp16 or bf16), enable gradient checkpointing, shorten sequence length, or use parameter-efficient fine-tuning like LoRA with a quantised base model.

   Show a sample answerHide the sample answer

   First I'd halve the batch size and use gradient accumulation so the effective batch stays the same. Then turn on bf16 mixed precision and gradient checkpointing. For fine-tuning a large language model, I'd use LoRA on a 4-bit quantised base model, which lets a 7B model fit on a single consumer-size GPU.

6. Explain how attention works in a transformer, in simple terms.

   Why they ask

   Checks you understand the architecture behind the models you deploy.

   How to answer

   Each token creates a query, key and value vector. Attention scores come from the dot products of queries with keys, scaled and passed through softmax, then used to take a weighted sum of values. This lets each token pull in information from the most relevant other tokens. Multi-head attention does this several times in parallel.

   Show a sample answerHide the sample answer

   In "the bank approved the loan", the word "bank" needs context from "loan" to know it's not a riverbank. Each token's query is compared with every token's key; the softmax of those scores says how much to weight each token's value. The result is a new representation of "bank" that includes information from "loan".

7. How would you build and evaluate a RAG system for customer support documents?

   Why they ask

   RAG is the most common LLM feature companies are building. Evaluation is where weak candidates fall down.

   How to answer

   Cover ingestion and chunking, embeddings and a vector index, retrieval (often hybrid keyword plus vector, with reranking), prompt construction with citations, and evaluation: a labelled question set measuring retrieval (did the right chunk come back?) and answer quality (correct, grounded in sources). Mention monitoring and feedback in production.

   Show a sample answerHide the sample answer

   I'd split the help articles into chunks of a few hundred tokens along headings, embed them and store them in pgvector, with keyword search alongside for exact terms like product codes. For each question I'd retrieve the top 20, rerank to the best 5 and ask the model to answer only from them with citations. Before launch I'd write 100 real customer questions with the right article, measure recall@5 for retrieval, then grade answers for correctness and grounding. Every change gets measured against that set.

8. When would you fine-tune a model instead of using prompting or RAG?

   Why they ask

   Fine-tuning is expensive and often unnecessary. They want sound judgement.

   How to answer

   Use RAG when the model needs facts that change or are private. Use prompting first for most behaviour. Fine-tune when you need a consistent format or style, a narrow task done cheaply by a smaller model, or behaviour prompting can't achieve, and only when you have good training data and an evaluation set.

   Show a sample answerHide the sample answer

   If the model needs to know our latest pricing, that's RAG, because fine-tuning won't keep up with changes. If we need thousands of support tickets classified into 30 categories cheaply, I'd consider fine-tuning a small open model on labelled tickets, after checking that a prompted model isn't already good enough on our evaluation set.

9. An LLM feature is too slow and too expensive. How would you improve it?

   Why they ask

   Inference cost is a big line item at AI startups. Practical ideas matter.

   How to answer

   Measure first: tokens in and out, latency per step. Then shorten prompts and context, cache repeated results, use a smaller model for easy cases (routing), batch requests, stream responses to cut perceived latency, and for self-hosted models use vLLM, quantisation and continuous batching.

   Show a sample answerHide the sample answer

   I'd log token counts and latency per request to see where the cost is. Often the prompt is stuffed with 20 retrieved chunks when 5 would do. I'd cache answers to frequent questions, send simple queries to a smaller model and only escalate hard ones, and stream the response so users see text straight away.

10. Write a function that returns the top k most similar vectors to a query using cosine similarity.

    Why they ask

    A small coding task that tests NumPy fluency and understanding of embeddings.

    How to answer

    Normalise the vectors, take dot products, then sort. Mention np.argpartition for large arrays and that vector databases do this approximately at scale.

    Show a sample answerHide the sample answer

    ```python
    import numpy as np

    def top_k_similar(query, matrix, k=5):
    q = query / np.linalg.norm(query)
    m = matrix / np.linalg.norm(matrix, axis=1, keepdims=True)
    scores = m @ q
    idx = np.argsort(scores)[::-1][:k]
    return idx, scores[idx]
    ```

    For millions of vectors I'd use np.argpartition to avoid a full sort, or an approximate index like HNSW in a vector database.

11. Design a fraud scoring system for card payments that must respond in under 100 milliseconds.

    Why they ask

    A classic ML system design question for fintechs, which are big hirers of ML engineers in and around Nigeria.

    How to answer

    Clarify requirements (volume, latency, cost of errors), then cover features (real-time and precomputed), a feature store or cache, a fast model such as gradient boosting, the serving path, fallbacks if the model is down, human review queues, monitoring and delayed labels from chargebacks.

    Show a sample answerHide the sample answer

    I'd precompute slow features like a customer's 30-day spending in a batch job and keep them in Redis, and compute fast ones like the number of attempts in the last 5 minutes from a stream. A LightGBM model scores in a few milliseconds behind a small service. Scores above one threshold block, a middle band goes to manual review, and if the service fails we fall back to simple rules rather than block everyone. Chargebacks arrive weeks later, so I'd track proxy metrics daily and true precision monthly.

Behavioural questions

Past situations. Answer with a real example: what happened, what you did, what changed.

1. Tell me about a time you chose a simpler model over a more complex one.

   Why they ask

   Pragmatism is highly valued in ML engineering.

   How to answer

   Explain the options, the trade-offs (accuracy, latency, cost, maintainability, explainability) and the outcome.

   Show a sample answerHide the sample answer

   For a loan pre-approval feature, a neural network beat logistic regression by about one point of AUC offline. But the credit team needed to explain every decision, and the simpler model ran in a millisecond on a small server. We shipped logistic regression with well-designed features. A year later it's still running, and the team understands it.

2. Tell me about a production incident involving a model you owned.

   Why they ask

   Everyone who has run models has a story. They want honesty and what you changed afterwards.

   How to answer

   Describe what happened, how it was detected, how you fixed it and what you changed so it couldn't happen silently again.

   Show a sample answerHide the sample answer

   A recommendation model started showing the same five products to everyone. An upstream change had made one feature null for all users, and the model fell back to popularity. We noticed from a drop in click-through two days later. I fixed the pipeline, then added input validation that rejects batches with too many nulls and an alert on prediction diversity.

Situational questions

Hypotheticals about problems you'd meet in the job.

1. A customer reports that the AI assistant gave a confidently wrong answer about their account. What do you do?

   Why they ask

   LLM features fail in public. They want responsible, systematic handling.

   How to answer

   Contain first (correct the customer, check if others were affected), reproduce with logs, find whether retrieval, the prompt or the model failed, add the case to the evaluation set, fix and measure, and add guardrails if needed.

   Show a sample answerHide the sample answer

   I'd make sure support corrects the customer straight away, then pull the logged request: the question, retrieved documents and response. If retrieval brought back an outdated policy, the fix is in the index; if the right document was there and the model ignored it, I'd look at the prompt or add a check that answers cite a source. The case goes into the evaluation set so we catch it if it ever comes back.

Remote work questions

How you'll work from Nigeria with a team elsewhere: hours, power, internet, communication.

1. How do you manage long training jobs and GPU costs when working remotely?

   Why they ask

   Remote ML engineers work entirely on cloud machines, and cost discipline matters.

   How to answer

   Mention running jobs on cloud instances not your laptop, checkpointing, using spot or preemptible instances where safe, tracking experiments, and shutting resources down automatically.

   Show a sample answerHide the sample answer

   Jobs run on cloud GPUs, never on my laptop, so a power cut in Lagos doesn't matter. I checkpoint regularly so I can use cheaper spot instances, log every run in MLflow or Weights & Biases, and use scripts that shut the instance down when the job finishes. I post a short summary of each experiment in our channel so the team can follow without a meeting.

2. What does your home setup look like for power and internet?

   Why they ask

   Employers hiring in Nigeria want reassurance about calls, on-call and access.

   How to answer

   Be concrete and honest about backup power and two internet sources.

   Show a sample answerHide the sample answer

   I run on solar with an inverter and batteries for my laptop, monitor and router, and I have two internet connections from different providers. All heavy work runs in the cloud. If both lines failed during an incident, I can still tether from my phone to check dashboards and roll back.

Remote interview tips

Questions people ask